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It is clear from the growing role of Ad Exchanges in the real-time sale of advertising slots that web publishers 
are considering a new alternative to their more traditional reservation-based ad contracts. To make this choice, 
the publisher must trade off, in real-time, the short-term revenue from an Ad Exchange with the long-term 
benefits of delivering good quality spots to the reservation ads. 

In this paper, we formalize this combined optimization problem as a stochastic control problem and derive 
an efficient policy for online ad allocation in settings with general joint distribution over placement quality 
and exchange prices. We prove asymptotic optimality of this policy in terms of any arbitrary trade-ofT between 
quality of delivered reservation ads and revenue from the exchange, and provide a rigorous bound for its 
convergence rate to the optimal policy. We also give experimental results on data derived from real publisher 
inventory, showing that our policy can achieve any Pareto-optimal point on the quality vs. revenue curve. 



1. Introduction 

Internet Display Advertising refers generally to the graphical and video ads that are now ubiquitous 
on the web. These types of ads generated about 10 billion dollars in the US in 2010, and analysts see a 

. Traditionally, an advertiser would buy display 
ad placements by negotiating deals directly with a publisher (the owner of the web page) , and signing 
an agreement, called a guaranteed contract. These deals usually take the form of a specific number of 
ad impressions reserved over a particular time horizon (e.g., one million impressions over a month). 
A publisher can make many such deals with different advertisers, with potentially sophisticated 
relationships between the advertisers' targeting criteria. The publisher would then need to assign 
arriving impressions to the matching reservations so as to maximize the placement quality of the 
contracts. Typically, the probability that a user clicks on an ad (known as click-trough rate) is used 
as a metric of placement quality. 

Guaranteed contracts can suffer in efficiency: since slots are booked in advance, both parties cannot 
react to instantaneous changes to traffic patterns or market conditions. However, this has changed in 
the last couple of years. Advertisers may now purchase ad placements through spot markets for the 
real-time sale of online ad slots, called Ad Exchanges. Prominent examples of exchanges are Yahoo's 
RightMedia, Microsoft's AdECN, Google's DoubleClick and OpenX. While exchanges differ in their 
implementations, in a generic Ad Exchange (AdX) 
with a reservation price, advertisers post bids, and an auction is run; this happens between the time 



clear rising trend (Internet Advertising Bureau||20TT 



Muthukrishnan||2009 ) , pubhshers post an ad slot 
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a user visits a page and the ad is displayed. Ad exchanges aUow advertisers to bid in real time and 
pay only for valuable customers, instead of bulk buying impressions and targeting large audiences. 

In presence of ad exchanges, publishers face the problem of maximizing the overall placement 
quality of the impressions assigned to the reservations together with the total revenue obtained 
with AdX, while complying with the contractual obligations. Note that these two objectives are 
potentially conflicting; in the short-term, the publisher might boost the revenue stream from AdX at 
the expense of assigning lower quality impressions to the advertisers. In the long term, however, it 
may be convenient for the publisher to prioritize her advertisers in view of attracting future contracts. 
So for a given piece of ad inventory, the publisher must quickly decide whether to send the inventory 
to AdX (and at what price), or to assign it to an advertiser with a reservation. 

In this paper, we study the problem faced by the publisher, jointly optimizing over AdX and the 
reservations. We bring to bear techniques from revenue management and stochastic optimal control, 
perform a probabilistic modeling of the problem, and derive an efficient policy for making real-time 
ad allocation decisions. We prove that our policy is asymptotically optimal in terms of an arbitrary 
(i.e., publisher-defined) trade-off between quality delivered to reservation ads and revenue from the 
exchange. Our policy and analysis is quite general, and works for any joint distribution over placement 
quality and exchange prices, even allowing correlation between advertisers, or between quality and 
exchange prices. In particular, we provide a rigorous bound on the convergence rate of our policy 
to the optimal policy (Theorem [2]) . Typically ad allocation research compares to the optimal offiine 
policy in hindsight; instead, we compare our policy with an optimal online policy, obtaining a bound 
on additive regret, as in online machine learning. 

Since the optimal policy cannot be computed efficiently in most real-world problems, we derive 
a provably good policy which resembles a bid-price control but extended with a pricing function to 
take into account for AdX. Our policy assigns each guaranteed contract a bid-price (or dual variable), 
which may be interpreted as the opportunity cost of assigning one additional impression to the 
advertiser. When a user arrives, the pricing function quotes a reserve price to submit to the exchange 
that depends on the opportunity cost of assigning the impression to an advertiser. If AdX price 
does not exceed this reserve price, the impression is immediately assigned to the advertiser whose 
placement quality exceeds its opportunity cost by the largest amount. We also give experimental 
results on data derived from real publisher inventory, showing that our policy can achieve any Pareto- 
optimal point on the quality vs. revenue curve. 

1.1. Related Work and Contributions 

Our works draws on three streams of literature, namely, that of Display Advertising, Revenue Man- 
agement, and Online Allocation. Rather than attempting to exhaustively survey the literature on 
each area, we focus on the work more closely related to ours. 

Display Advertising. There has been recent work on display ad allocation with both contract-based 



advertisers and spot market advertisers. Ghosh et al. (2009) focus on "fair" representative bidding 



strategies in which the publisher bids on behalf of the contract-based advertisers competing with the 
spot market bidders. This line of work is mainly concerned with computing such fair representative 
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bidding strategies for contract-based advertisers. Chen (2011) considers the case when the publisher 
runs the exchange, and employing a mechanism design approach he characterizes, through dynamic 
programming, the optimal dynamic auction for the spot market. In this model both bids from the 
spot market and the total number of impressions are stochastic. We focus, instead, on combined 
yield optimization and present a model and an algorithm taking into account any trade-off between 
quality delivered to reservation ads and revenue from the spot market. 



Yang et al. (2010) studied the problem faced by the publisher of allocating between the two 



markets using multi-objective programming. As in our work, they consider different objectives for the 
publisher, such as, minimizing the penalty of under-delivery, maximizing the revenue from the spot 
market and the representativeness of the allocation. However, they employ a deterministic model with 
no uncertainty in which future inventory and contracts are nodes in a bipartite graph. Alaei et al. 



(2009) proposed an utility model that accounts for two types of advertisers: one oriented towards 
campaigns and seeking to create brand equity, and the other oriented towards the spot market and 
seeking to transform impressions to sales. Here impressions are commodities which can be assigned 
interchangeably to any advertisers. In this setting they look for offline and online algorithms aiming to 



maximize the utility of their contracts of the allocation. Roels and Pridgeirsdottir ( 2009 ) studied the 
scheduling problem in display advertising in the case without the exchange. In this paper the publisher 
needs to decide, as new contracts arrive, whether to accept them or not, and then dynamically deliver 
arriving impressions to them. They take into account uncertainty both in supply and demand, provide 
a dynamic programming formulation, and propose a certainty-equivalent control. 



Revenue Management. Another stream of relevant work is that of Revenue Management (RM). 



Even though RM is typically applied to airlines, car rentals, hotels and retailing (Talluri and van 



Ryzin 2004), our problem formulation and analysis is inspired by RM techniques. As in the pro- 



totypical RM problem, we look for a policy maximizing the ex-ante expected revenue, which can 
be obtained using dynamic programming (DP). Since the resulting DP is intractable, we aim for a 
deterministic version in which stochastic quantities are replaced by their expect values and quantities 



assumed to be continuous. These are common in the literature (Gallego and van Ryzin 1994, Liu 



and van Ryzin 2008), and provide policies with provably good performance. Indeed, we show that 



our policy is asymptotically optimal. 

The Display Ad problem can be thought of as a parallel-flight Network RM problem (see, e.g.. 



Talluri and van Ryzin (1998)) in which users' click probabilities are requests for itineraries, and 



advertisers are edges in the network. The are three differences, however, with the traditional Network 
RM problem. First, we aim to satisfy all contracts, or completely deplete all resources by the end 
of the horizon. Second, in the traditional problem requests are for only one itinerary (which can 
be accepted or rejected), while in our model each impression can be potentially assigned to any 
contract and the publisher needs to decide whom to assign the impression based on possibly correlated 
placement qualities. Finally, the publishers in display advertising may submit impressions to a spot 
market to increase their revenues. 
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Decide whom to assign to 
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No 
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Table 1 Comparison of the Display Ad and Network Revenue Management problems. 



A popular method for controlling the sale of inventory in revenue management applications is 



the use of bid-price controls. These were originally introduced by Simpson (1989), and thoroughly 



analyzed by Talluri and van Ryzin (1998). In this setting, a bid-price control sets a threshold or bid 
price for each advertiser, which may be interpreted as the opportunity cost of assigning one additional 
impression to the advertiser. This approach is standard in the context of revenue maximization. 



e.g. the stochastic knapsack problem by Levi and Radovanovic (2010). From this perspective, our 
contribution is the inclusion of a spot market, the exchange, as an new sales channel. In this case, our 
policy is a suitable modification of a bid-price control that takes into account AdX by incorporating a 
pricing function. This pricing function quotes a reserve price to submit to the exchange that depends 
on the opportunity cost of assigning the impression to an advertiser. 

Online Allocation. Our work is closely related to online ad allocation problems, including the 



Display Ads Allocation (DA) problem (Feldman et al. 2009 2010 Agrawal et al. 2009 Vee et al. 



2010| ), and the AdWords (AW) problem ( |Mehta et al.|2007[|Devenur and Hayes|2009D . In both of these 



problems, the publisher must assign online impressions to an inventory of ads, optimizing efficiency 
or revenue of the allocation while respecting pre-specified contracts. 

In the DA problem, advertisers demand a maximum number of eligible impressions, and the pub- 
lisher must allocate impressions that arrive online to them. Each impression has a potentially different 
value for every advertiser. The goal of the publisher is to assign each impression to one advertiser max- 
imizing the value of all the assigned impressions. The adversarial online DA problem was considered 
in Feldman et al. ( 2009 ) , which showed that the problem is inapproximable without exploiting free 



disposal; using this property (that advertisers are at worst indifferent to receiving more impressions 
than required by their contract), a simple greedy algorithm is ^-competitive, which is optimal. When 
the demand of each advertiser is large, a (1 — ^)-competitive algorithm exists ( Feldman et al.|2009 ), 
and it is tight. The stochastic model of the DA problem is more related to our problem. Following 
a training-based dual algorithm by Devenur and Hayes (2009), training-based (1 — e)-competitive 
algorithms have been developed for the DA problem and its generalization to various packing linear 



programs ( [Feldman et al.||2010| |Vee et al.||2010| [Agrawal et al.||2009[ ). 

In the AW problem, the publisher allocates impressions resulting from search queries. Here each 
advertiser has a budget on the total spend instead of a bound on the number of impressions. Other 
than training-based dual algorithms and primal-dual algorithms that get similar bounds as in the 



DA problem (Devenur and Hayes 2009), online adaptive optimization techniques have been applied 
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to online stochastic ad allocation (Tan and Srikant 2010). Such control-based adaptive algorithms 



achieve asymptotic optimality following an updating rule inspired by the primal-dual algorithms. 

Our work differs from all the above in three main aspects: (i) We study both the parametric and 
non-parametric models, and compare their effectiveness in terms of the size of the sample sizes — 
both analytically for various distributions and experimentally on real data sets, (ii) Instead of using 
the framework of competitive analysis and comparing the solution with the optimum solution in 
hindsight, we compare the performance of our algorithm with the optimal online policy, and present 
a rate of convergence bound under this model. This is akin to regret bounds found in online Machine 
Learning; and (iii) None of the above work considers the simultaneous allocation of reservation 
ads and ads from AdX. In particular, these previous works do not consider the trade-off between 
the revenue from a spot market based on real-time bidding and the efficiency of reservation-based 
allocation. 

It is tempting to simply reduce to online stochastic packing by considering the AdX as just another 
"advertiser." The problem with this is that it does not allow adjusting the reserve price, or allocating 
a reservation advertiser if the AdX rejects. In fact one can make such a reduction go through by 
considering online decisions on pairs of (reserve price, advertisers), and formalize the problem as an 
online allocation problem with general packing constraints. After a couple more steps, one can apply 



the techniques of Devenur and Hayes (2009), Feldman et al. (2010), Vee et al. (2010), Agrawal et al. 



(2009) to derive an online algorithm for the combined problem. However this approach discretizes 
the price space into multiples of 6; thus (a) we lose 1 in the yield, (b) we increase the running time 
by a factor of |, and (c) for the the competitiveness proof to hold, we need more stringent conditions 
on the size of the weights. In addition, there is no clear way to apply the parametric technique. The 
method presented in this work not only avoids this dependence, it is a much more natural, extensible 
solution to the problem. 

2. Model 

Consider a publisher displaying ads in a web page. The web page has a single slot for display ads, 
and each user is shown at most one impression per page. The publisher has signed contracts with a 
set A of advertisers guaranteeing them a certain number of targeted impressions within a given time 
horizon. We denote by ^ = {1, . . . , A} the set of advertisers. 

Even though the number of users visiting a web page is uncertain, publishers usually have fairly 
good estimates of the total number of expected users that arrive in a given horizon. In this model 
we index time based on the arrival of each user, and assume that the total number of users is fixed 
and equal to A'^. We do allow users to have different characteristics (random number of users can be 
accommodated in our model by considering dummy arrivals). Indeed, depending on the user profile, 
the impression may be more or less attractive for different advertisers. 

We assume that the n-th impression is endowed with a vector of placement qualities Q„ = 
{Qn,a}aeAi whcrc Qn,a IS the predicted quality advertiser a would perceive if the impression is assigned 
to her. Qualities lie in some compact space O C M^. A typical measure of placement quality is the 
estimated probability that the user click on each ad. In practice, such measure of quality is learned 
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Impression n-th 
arrives with 
quality Q„ 




Assign to an advertiser or discard 



Submit to AdX 
with price p 




Figure 1 Publisher's decision tree for a new impression. 



Obtain payment 



Assign to an 
advertiser or 
discard 



using, for example, logistic regression. Here we abstract from the learning problem and assume that 
qualities {Qn}n=i,...,N are random and drawn independently from some joint c.d.f G(-). We do allow, 
however, for qualities to be jointly distributed across advertisers. This captures the fact that adver- 
tisers might have similar target criteria, and hence the qualities perceived might be correlated. We 
do not impose any further restrictions on the qualities, other than finite second moments. Notice 
that the publisher observes the realization of the placement quality before showing the ad. 

The publisher has agreed to deliver exactly Ca impressions to advertiser a£ A; neither over-delivery 
nor under-delivery is allowed. We denote hy p = {pa}aeA with Pa = ^-, the capacity to impression 
ratio of each advertiser. Note that a necessary condition for the feasibility of the operation is that 
the number of arriving impressions suffices to satisfy the contracts, or Xlae^/^a — ^- assumption 
of this general model is that any user can be potentially assigned to any advertiser. In practice each 
advertiser may be interested in a particular group of user types. It is important to note that this is 
not a limitation of our results, but rather a modeling choice; in ^we show how to handle targeting 
criteria by setting Qn,a = —Ta for impressions not matching the targeting criteria of an advertiser. 
This can also be interpreted as forcing the publisher to pay a good will penalty to the advertisers 
each time an undesired impression is incorrectly assigned. 

Arriving impressions may either be assigned to the advertisers, discarded or auctioned in the Ad 



Exchange (AdX) for profit. In a general AdX ( Muthukrishnan 2009), the publisher contacts the 
exchange with a minimum price she is willing to take for the slot. Additionally, the publisher may 
submit some partial information of the user visiting the website. For simplicity, we first assume that 



no information about the user is revealed. However, in section 7.3 we relax this assumption. Internally 
the exchange contacts different ad networks, and in turn they return bids for the slot. The exchange 
determines the winning bid among those that exceed the reserve price via an auction, and returns 
a payment to the publisher. In this case we say that the impressions is accepted, and the publisher 
is contractually obligated to display the winning impression. In the case that no bid attains the 
reserve price, no payment is made and the impression is rejected. We present the formal model of the 



exchange in Section 2.2 The entire operation above is executed before the page is rendered in the 
user's screen. Thus, in the event that the impression is rejected by the exchange, the publisher may 
still be able to assign it to some advertiser. Figure [T] summarizes the decisions involved. 
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For notational simplicity we extend the set of advertisers to = {0} by including an outside 
option that represents discarding an impression. We set the quality of the outside option identically 
to zero, i.e. Qn,o = for all impressions n = 1, . . . , N . In the following, the terms discarding an 
impression or assigning it to advertiser are used interchangeably. We set po = l — ^^^eAP'^ 
fraction of impressions that are not assigned to any advertiser. To wit, a fraction of the po impressions 
will be assigned to the winning impression of AdX, and the remainder effectively discarded. 

2.1. Objective 

The publisher's problem is to maximize the overall placement quality of the impressions assigned to 
the advertisers together with the total revenue obtained with AdX, while complying with the contrac- 
tual obligations. Note that the objectives are potentially conflicting; in the short-term, the publisher 
might boost the revenue stream from AdX at the expense of assigning lower quality impressions to 
the advertisers. In the long term, however, it may be convenient for the publisher to prioritize her 
advertisers, in view of attracting future contracts. 

We attack the multi-objective problem by taking a weighted sum of both objectives. The publisher 
has at her disposal a parameter 7, which allows her to trade-off between these conflicting objectives. 
The aggregated objective is given by 

yield = revenue(AdX) -|- 7 • quality (advertisers). 

Hence, by choosing a suitable large 7 the advertisers may focus on assigning high quality impressions 
to the advertisers; while a small 7 would prioritize the revenue from AdX (the publisher may set 
different values of the parameter for each advertiser) . Without loss of generality, we set 7 = 1 for the 
remainder of this paper, except when noted otherwise. 

Alternatively, the publisher might impose that the overall quality of the impressions assigned to 
the advertiser is greater than some threshold, and then maximize the total revenue obtained from 
AdX; this may have a more natural interpretation for some publishers, and would be simpler than 
having to set 7. We can model this simply by interpreting 7 as the Lagrange multiplier of the quality 
of service constraint, and our problem as the Lagrange relaxation of the constrained program. In §7.1| 
we analyze the implications of this formulation, and in ^6.1 we study experimentally the impact of 
the choice of 7 on both objectives. 

2.2. AdX Model 

The publisher submits an impression to AdX with the minimum price it is willing to take, denoted 
by p > 0. The impression is accepted if there is a bid of value p or more. We denote by B the winning 
bid random variable. In the following we assume that bids are independent of the quality of the 
impression, and identically distributed according to a c.d.f. F{-). Hence, the impression is accepted 
with probability 1 — F{p) = F{p). In this first model, when the impression is accepted, the publisher 



is paid the minimum price p. In Sections 7.2 and 7.3 we drop this assumption and consider a more 
general second-price auction with side information. 

Suppose the publisher has computed an opportunity cost c for selling this inventory in the exchange; 
that is, the publisher stands to gain c if the impression is given to a reservation advertiser. Given 
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opportunity cost c > the publisher picks the price that maximizes its expected revenue. Hence, the 
pubhsher solves the optimization problem R{c) =maXp>o-F(p)p + F(p)c. Changing variables, we can 
define r(s) = sF^^{s) to be the expected revenue under acceptance probability s, and rewrite this as 

R(c) = max r(s) + (1 — s)c. (1) 

se[o,i] 

Also, let s*{c) be the least maximizer of ([T]), and p*{c) = F^^ {s*{c)) be the price that verifies the 
maximum. 

Assumption 1 The expected revenue under survival probability s is continuous, concave, non- 
negative, bounded, and satisfies lims_^Qr{s) = 0. We call a function r{s) that satisfies all of the 
assumptions above a regular revenue function. 



These assumptions are common in RM literature (see, e.g., Gallego and van Ryzin (1994)). A 
sufficient condition for the concavity of the revenue is that B has increasing generalized failure 
rates ( Larivierel|2006 ) . Regularity implies, among other things, the existence of a null price p^o such 
that limp_j.p^ F{p)p = 0. Additionally, it allows us to characterize the value function R[c). In { 7.2 we 
show that r(s) remains regular in the presence of multiple bidders in the AdX by considering the 
joint density of the highest and second-highest bids. Thus, all our results hold in this case too. 

Proposition 1 Suppose that r{s) is regular revenue function. Then, R{c) is non- decreasing, convex, 
continuous, and R{c) > c. Additionally, R{c)—c is non-increasing, s*(c) is non-increasing, andp*{c) 
is non- decreasing. 

An important consequence of above is that the maximum revenue expected from submitting an 
impression to AdX is always greater than the opportunity cost. This should not be surprising, since 
the publisher can pick a price high enough to compensate for the revenue loss of not assigning the 
impression. Hence, assigning an impression directly to an advertiser (rather than first testing the 
exchange) is never the right decision, and so in Figure [l]the upper branch is never taken. 

3. Problem Formulation 

In this section we start by formulating an optimal control policy for yield maximization based on 
dynamic programming (DP), where the state of the system is represented by the number of impres- 
sions yet to arrive, and a vector of the number of impressions needed to comply with each advertiser's 
contract. Unfortunately, the state space of the DP has size 0{N^^^), and in most real-world prob- 
lems the number of impressions in a single horizon can be in the order of millions. So the DP is not 
efficiently solvable. We give, instead, an approximation in which stochastic quantities are replaced 
by their expected values, and are assumed to be continuous. Such "deterministic approximation 



problems (DAP)" are popular in RM (see, e.g., Talluri and van Ryzin (1998)). In our setting, the 



approximation we make is to enforce contracts to be satisfied only in expectation. We formulate the 
problem based on this assumption and obtain an infinite-dimensional program. This DAP is solved 
by considering its dual problem, which turns out to be a more tractable finite-dimensional convex 
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program. Finally, we wrap a full stochastic policy around it (one that always meets the contracts, 
not just in expectation). 

3.1. Dynamic Programming Formulation 

Let {m,X) be the state of the system, where we denote by m the total number of impressions 
remaining to arrive, and hy X = {xa}aeA the number of impressions needed to comply with each 
advertiser's contract. Let the value function, denoted by J,n{X), be defined as the optimal expected 
yield obtainable under state {m,X). Using the fact that is optimal to first test the exchange, we 
obtain the following Bellman equation 



J„(X) = Eq„ [max + J„_i(X)) + (1 - F{p)) max{Q„,, + J„_i(X - Ij}) 

L p>0 L aeAo J 

= J„_i(X) +Eq„ R ( max{Q„.„ - A, J„_i(X)} ) 



where we defined 1^ as a vector with a one in entry a and zero elsewhere, Ig = 0, and Jm(X) = 
Jm{X) — Jm{x — la) ss the cxpccted marginal yield of one extra impression for advertiser a. In ^ 
the objective accounts for the yield obtained from attempting to send the impression to AdX. In the 
yield has two terms that depend on whether the impression is accepted or not by AdX. In the latter 
case the maximum accounts for the decision of assigning the impression directly to the advertiser or 
discarding the impression (when a = 0). In ^ we used the fact that assigning an impression directly 
to an advertiser is never the right decision (except in boundary conditions, see below). The publisher, 
however, may choose to discard impressions with low quality after being rejected by AdX. 

Our objective is to compute J*^ = Jn{C). Let M be an upper-bound on the expected yieldj^The 
boundary conditions are 

J,„(x) = — M, \/X s.t. Xa < for some a£ A, 

J,n{x) = -M, \/m<^Xa. 

aeA 

Recall that when the contract with an advertiser is fulfilled, no extra yield is obtained from assigning 
to her more impressions. This is the case of the first boundary condition, which guarantees that 
advertisers whose contract is fulfilled are excluded from the assignment. In particular, when X = all 
remaining impressions are sent to AdX with the yield maximum price p*{0) when x = 0. The second 
boundary condition guarantees that the contracts with the advertisers are always fulfilled. When 
^^g^Za = m AdX must be bypassed, and impressions should be assigned directly to the advertisers. 
The optimal policy is described in Policy [T} 

In the above policy, when the impression is submitted to AdX, the optimal price ponders an 
opportunity cost of Qn,a* — A^* J„_i(X). This opportunity cost, when positive, is just the value 
of the impression adjusted by the loss of potential yield from assigning the impression right now. 
Note that the two boundary conditions are implicit in the optimal policy. This guarantees that the 
policy complies with the contracts. It is routine to check that the value function Jn{X) is finite for 

^ One could set, e.g., M = max{poo, Q} where Q is an upper-bound on the placement quality 
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Policy 1 Optimal dynamic programming policy. 
Observe state {m,X) and the realization Q„. 

Let < = argmaxae^o {QnM - Aa J„-i(X)}. 

Submit to AdX with price p* (Qn,a* — A^* J„_i(X)). 

if impression rejected by AdX and a* 7^ then 

Assign to advertiser a* . 
end if 



all feasible states and that Policy [I] is optimal for the dynamic program in ([2]) . It is worth noting 
that in order to implement the optimal policy one needs to pre-compute the value function, which is 
intractable in most real instances. 

3.2. Deterministic Approximation Problem (DAP) 

We aim for an approximation in which (i) the policy is independent of the history but dependent on 
the realization of Qn, (ii) capacity constraints are met in expectation, and (iii) controls are allowed 
to randomize. These approximations turn out to be reasonable when the number of impressions is 
large. When an impression arrives, the publisher controls the reserve price submitted to AdX, and the 
advertiser to whom the impression is assigned, if rejected by AdX. Alternatively, in this formulation 
we state the controls in terms of total probabilities, where each control is a function from the quality 
domain to [0,1]. Let s = {s„(-)}n=i,...,Ar and i = {ini-)}n=i....,N be vectors of functions from to 
M, such that when the n^^ impression arrives with quality Q the impression is accepted by AdX 
with probability s„(Q), and with probability in,a{Q) it is assigned to advertiser a. The conditional 
probability of an impression being assigned to advertiser a given that it has been rejected by AdX 
is given by In,a{Q) = ^n.a(Q)/(l — Sn{Q))- When it is clear from the context, we simplify notation by 
eliminating the dependence on Q from the controls. 

A control is feasible for the DAP if (i) it satisfies the contractual constraint in expectation, (ii) the 
individual controls are non-negative, and (iii) for every realization of the qualities the probabilities 
sum up to at most one. We denote by V the set of controls that satisfy the latter two conditions. 
That is, V = {{s,i) : '^^aeA ia + s <l,s>^,i>Q}. The objective of the DAP is to find a sequence of 
real-valued measurable functions that maximize the expected yield, or equivalently 



The first term of the objective accounts for the revenue from AdX, while the second accounts for 
the quality perceived by the advertisers. Notice that in the DAP we wrote the total capacity as N pa 
instead of Cq to allow the problem to be scaled. 

Alas, the problem is still hard to solve since the number of functions is linear in N . However, 
exploiting the regularity of the revenue function, we can show that in the optimal solution to DAP, 




N 




(3a) 
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we can drop the dependence on n in the controls. This follows from the linearity of the constraints 
together with the concavity of the objective. We formalize this discussion in the following proposition. 

Proposition 2 Suppose that the revenue function is regular. Then, there exists a time-homogenous 
optimal solution to the DAP, i.e. where SniQn) = s{Qn) for all n = l,. . . ,N and in{Qn) = i{Qn) for 
alln=l,...,N. 

The previous proposition allows us scale the problem so that A'^ = 1, and consider the maximum 
expected revenue of one impression, denoted by . The total revenue for the whole time horizon 
is then = NJ^. In order to compute the DAP's optimal solution, we consider its dual problem, 
which we informally derive next. 

Derivation of the Dual to DAP. To find the dual, we introduce Lagrange multipliers v = {va}aeA 



for the capacity constraints (3a). The Lagrangian, denoted by C{s,i;v) is 



C{s, z; f ) = E 



aeA 



aeA 



The dual function, denoted by il^{v), is the supremum of the Lagrangian over the set V. Thus, we 
have that 



■ sup /l{s,i]v) 

(s,i)ev 

■ sup < E [r{s)] + sup 

''>0 I i>»-T.aeAia<l- 



E 



y^ja{Qa-Va) 



aeA 



+ ^ VaPa 



aeA 



■ supE 



s>0 



r(s) + (1 - s) max{Qa - Va} 

aeAo 



+ ^ VaPa 



aeA 



Ei2( max{QQ - Va}) + VaPa 
aeA^ ^ — ' 



aeA 



where the first equation follows from partitioning the optimization between the AdX acceptance and 
the assignment probability controls, the second from optimizing over the advertiser assignment con- 
trols i, and the last equation from solving the AdX variational problem. Note that R is convex and 
non-decreasing and the maximum is convex w.r.t v, hence the composite function within the expec- 
tation is convex. Using the fact that expectation preserves convexity, we obtain that the objective 
ip{v) is convex in v. 

Next, the dual problem is min„ V'(^)- When the revenue function is regular, the DAP's objective is 
concave and bounded from above. Moreover, the constraints of the primal problem are linear, and the 



feasible set V convex. Hence, by the Strong Duality Theorem (p. 224 in Luenberger (1969)) the dual 



problem attains the primal objective value. So, we have that dual problem is given by the following 
convex stochastic problem 



Jf = min<^ Ei?(max{Qa - Va]) + VaPa \ 

V I aeAn ^ — ' 



(4) 
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Deterministic optimal control. When the distribution Q is known, the dual problem in Q can 
be solved using a Subgradient Descent Method. It is worth noting that in many applications the 
distribution of Q is unknown and should be learned as impressions arrive. We postpone the discussion 
of that problem until §6.2[ 

Once the optimal dual variables v are known, the primal solution can be constructed from plugging 
the optimal Lagrange multipliers in C{s,i;v). Following the derivation of the dual, we obtain that 
the optimal survival probability is s{Q) = s* {maXaeAo{Qa — Va})- Hence, the impression has a value 
of maXag^glQa — Va} for the publisher, and she picks the reserve price that maximizes her revenue. 
From the optimization over the assignment controls, we see that an impression is assigned to an 
advertiser a only if she maximizes the contract adjusted quality Qa — Va- Thus the dual variables Va 
act as the bid-prices of the guaranteed contracts. Additionally, the impression can be discarded only 
if the maximum is not verified by an advertiser (i.e. all contract adjusted qualities are non-positive). 

Notice that optimizing the Lagrangian states that the impression should be assigned to an adver- 
tiser maximizing the contract adjusted quality, but does not specify how the impression should be 
assigned when -multiple- advertisers attain the maximum. In the case when the probability of a 
tie occurring is zero, the problem admits a simple solution: assign the impression to the unique 
maximizer of Qa — Va- We formalize this discussion in the the following theorem. 

Theorem 1 Suppose that the revenue function is regular, and there is zero probability of a tie occur- 
ring, i.e. P{Qa — Va = Q'a ~ v'a} = for all distinct a, a' £ Aq. Then, the optimal controls for the DAP 
are s{Q) = s* {uiaXaeAoiQa — Va}), and Ia[Q) = 1 {Qa — Va> Qa' — Va' Va' G Aq}, that is, the impres- 
sion is assigned to the unique advertiser maximizing the contract adjusted quality. Furthermore, the 
optimal dual variables solve the equations 

E[il-S*{Qa-Va))IaiQ)]=Pa, Vo G A 

3.3. Our Stochastic Policy 

The solution of the DAP suggests a policy for the stochastic control problem, but we must deal with 
two technical issues: (i) when more than one advertiser maximizes Qa — Va we need to decide how to 
break the tie, and (ii) we are only guaranteed to meet the contracts in expectation, whereas we must 
meet them exactly. We defer the first issue until §3.4[ where we give an algorithm for generalizing 
the controls Ia{Q) to the case where ties are possible. 

We propose a static bid- price control extended with a pricing function for AdX given by p* . The 
policy, which we denote by fi^ , is defined in Policy [2| In there we let Xn.a be the total number of 
impressions left to assign to advertiser a to comply with the contract, m = N — n the total number 
of impressions remaining to arrive, and v to be the optimal solution of Q. 

Notice that impressions are only assigned to advertisers with contracts that have yet to be fulfilled. 
When all contracts are fulfilled, impressions are sent to AdX with the revenue maximizing price p*{0). 
Moreover, when the total number of impressions left is equal to the number of impressions necessary 
to fulfill the contracts, the price is set to poo, and thus all incoming impressions are directly assigned 
to advertisers. Hence, the stochastic policy fi^ satisfies the contracts with probability 1. 
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Policy 2 Static Bid-Price Policy with Pricing fi^ . 
Observe state {m,X) and the reahzation Q„. 
Let An = {a£ A : Xn,a > 0} be the set of ads yet to be satisfied. 
Let o; = argmaxae^„u{0} {Qn^ - Va}- 

if Eae^A^"," < ^ ^^^^ Pn=P*{Qn.a*„ -Va*„), elsC Ict Pn = Po< 

Submit to AdX with price p„. 

if impression rejected by AdX and a* 7^ then 

Assign to advertiser a* . 
end if 



The proposed stochastic pohcy shares some resemblance with the optimal dynamic programming 
policy. The intuition is that, when the number of impressions is large, the actual state of the system 
becomes irrelevant because AaJm-iix) is approximately constant (for states in likely trajectories), 
and equal to Va- In that case both policies are equivalent. 

3.4. Handling ties 

Theorem [1] had an assumption that there would be no ties between advertisers verifying the maximum 
Qa — Va- In this section we show how to construct a primal optimal solution to the DAP and the 
corresponding stochastic policy in the general case (for example, when the distribution of placement 



quality is discrete or has atoms). Devenur and Hayes (2009) proposed introducing small random 



and independent perturbations to the qualities, or smoothing the dual problem to break ties. We 
provide an alternate method that directly attacks ties, and provides a randomized tie-breaking rule. 
Computing the parameters of the tie-breaking rule requires solving a flow problem on a graph of size 



2'^'; thus in some settings it may not be possible. In section EC. 3, we show that in practice ties do 
not occur frequently. However, for completeness we provide a full characterization of the problem. 

For any non-empty subset S C ^q, we define a S-tie as the event when the maximum is verified 
exactly by all the advertisers aG S, and the impression is rejected by AdX. Note that the tie may be 
a singleton, in the case that exactly one advertiser verifies the maximum. Since the dual variables v 
are known, the probability of such event can be written as 

P(5-tie) = K [(1 - s*{XiQ))l{Qa -Va = A(Q) Va G 5, - < \{Q) Va i S] 

where A(Q) = mayiaeAo{Qa — Va}- With some abuse of notation we define the %-tie as the event when 
the impression is accepted by AdX, that is, P(0-tie) = E[s*(A(Q))]. Note that the tie events induce a 
partition of the quality space, and we have that P(5'-tie) = 1. 

We look for a random tie-breaking rule that assigns an arriving impression to advertiser a£ S with 
conditional probability Ia{S) given that a S'-tie occurs. Hence, the routing probabilities depend on 
which advertisers tie, and not on the particular realization of the qualities (they are independent of 
X{Q)). Therefore, under such policy the total probability, originating from S-ties, of an impression 
being assigned to advertiser a is VaiS) = F{S-tie)Ia{S). We can interpret UaiS) as the normalized fiow 
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of impression assigned to the advertiser originating from S'-ties. We wih show that, in terms of ya{S) 
as decision variables, finding the tie-breaking rule amounts to solving a transportation problem. 

First, in order for the publisher to fulfill the contract with an advertiser a£ A the incoming flow 
of impressions over all possible ties sums up to pa- The previous constraint can be written as 

J] ya{S)=Pa, yaeA. (5) 

SCAo-.aeS 

Notice that we impose no constraints for a = since any number of impressions can be discarded. 
Alternatively, we could set p^^ = 1 — P(0-tie) — "^^eA Pa because the impressions effectively discarded 
are those that are rejected by AdX and not assigned to an advertiser. Second, the outgoing flow 
of impressions originating from a particular tie should sum up to the actual probability of that tie 
occurring. Then, we have that 

J]y,(5)=P(5-tie), (6) 

aes 

Third, we require that ya{S) > for all S C and aG S. Finally, in order to obtain the tie-breaking 
rule we need a non-negative flow satisfying constraints ^ and Once a such solution is found, 
the optimal controls can be computed as 



UQ) 



ya{S)/F{S-tie) if a G 5, and Q is an S-tie, 
otherwise. 



with the pricing function as before. 

It is not hard to see that the previous problem can be stated as a feasible flow problem in a bipartite 
graph. We briefly describe how to construct such graph next. On the left-hand side of the graph we 
include one node for each non-empty subset S C ^q, and in the right-hand side we add one node for 
each advertiser aG Aq- In the following we refer to nodes in the left-hand side as subset nodes, and 
to those in the right-hand side as advertiser nodes. The supply for subset nodes is P(5-tie), while 
the demand for advertiser nodes is pa- Arcs in the graph represent the membership relation, i.e., the 
subset node S and advertiser node a are connected if and only if a G 5. Moreover, arc capacities are 
set to infinity. In Figure [2] the resulting bipartite graph is shown. 

An important question is whether the flow problem admits a feasible solution. The next result 
proves that the answer is affirmative when the dual variables v are optimal for the dual problem 
Q. The proof proceeds by casting the feasible flow problem as a maximum flow problem, and then 
exploiting the optimality conditions of v to lower bound every cut in the bipartite graph. 

Proposition 3 Suppose that v € M"^ is an optimal solution for the dual problem Q . Then, there 
exists a non-negative flow satisfying constraints ^ and ([g]) . 

We conclude this section by showing that the solution constructed is optimal for the primal prob- 
lem. Notice that the solution is feasible because it satisfies constraints ([s]) and ([6]). In order to 
prove optimality it suffices to show that it attains the dual objective value, or that it satisfies the 
complementary slackness conditions. The latter follows trivially. 
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Subset nodes Advertiser nodes 

Figure 2 Bipartite flow problem solved to obtain the tie-breaking rule. On the left-hand side of the graph we include 
one node for each non-empty subset SQAo (subset nodes), and in the right-hand side we add one node 
for each advertiser aGAo (advertiser nodes). The supply for subset nodes is P(5-tie), while the demand 
for advertiser nodes is pa- Arcs in the graph represent the membership relation, i.e., the subset node S 
and advertiser node a are connected if and only if a€ S. Arc capacities are set to infinity. 

Once the optimal controls are calculated, we construct our stochastic policy as follows. We let 
= argmaxa£_4„u{0} {Qrt,a — ■^a}; be the set of advertisers that attain the maximum. Now, if the 
impression is rejected by AdX and 7^ {0}, we assign it to advertiser a in with probability 
Ia{Qn)/Yla'eA* ^a'{Qn)- Noticc that imprcssious are only assigned to advertisers with contracts that 
have yet to be fulfilled. Additionally, as the contracts of some advertisers are fulfilled, these are 
excluded of the assignment, and the routing probabilities Ia{-) oi the remaining advertisers are scaled- 
up and normalized. 

4. Asymptotic Analysis 

In this section we show that the heuristic policy constructed from the DAP is asymptotically optimal 
for the stochastic problem when the number of impressions and capacity arc scaled up proportionally. 
We proceed in the following way. First, we formulate the problem as a stochastic control problem 
(SCP). Though not practical, this abstract and equivalent formulation is useful from a theoretical 
point of view. Second, we show that the optimal objective value of the DAP provides an upper bound 
on the objective value of the SCP. Finally, we show that the upper bound is asymptotically tight. 

Stochastic Control Problem. A stochastic control policy maps states of the system to control actions 
(prices and target advertiser) , and is adapted to the history up to the decision epoch. We restrict our 
attention to policies that always submit the impression to AdX, which were argued to be optimal. 
Recall that given the reserve price, the publisher knows the actual probability that the impression 
is accepted by AdX. As before, we recast the problem in terms of the survival probability control. 
Hence, the publisher picks the probability that the impression is accepted. Conversely, given a survival 
probability the reserve price can be easily computed using F~^{-). We denote by G [0,1] the 

target survival probability under policy ji at time n when an impression with quality Q arrives. 
Similarly, we let InaiQ) ^ {0; 1} indicate whether the n*** impressions is assigned to advertiser a or 
not when policy /i is used. In particular, /^^(Q) = 1 indicates that the impression should be assigned 
to the advertiser if rejected by AdX. 
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We let the binary random variable X„(s^) indicate whether the n^^ impression is accepted by 
AdX or not when policy fi is used. Specifically, X„(s^) = 1 indicates that the impression is accepted 
by AdX, and when X„(s{^) = the impression is rejected by AdX. Notice that, conditioning on 
the quality of the impression and the history, X„(s^) is a Bernoulli random variable with success 
probability s^. 

We denote by Ai the set of admissible policies, i.e. policies that are non-anticipating, adapting 
and feasible. A feasible policy should satisfy the contractual obligations with each advertiser, or 
equivalently "^^=1 ~ -^n{sf^)] In.a = ^a in an almost sure sense. Additionally, the target advertiser 
controls should satisfy that J^aeA-^na — since the impression should be assigned to at most one 
advertiser. Finally, the equivalent stochastic optimal control problem is 



JZ, = max E 



N 



aeA 



(7) 



where denotes the optimal expected revenue over the set of admissible policies Ai. The objective 
follows from conditioning on the quality of the impression and the history. By the Principle of 
Optimality it is the case that the dynamic program described in section |3.1| provides an optimal 



solution to the SCP (Bertsekas 2000) and Jn{C) = J*^. 



Analysis. Following a similar analysis to Gallego and van Ryzin (1994), Talluri and van Ryzin 



(1998), Liu and van Ryzin (2008), we first show that the optimal objective value of the DAP provides 



an upper bound to the objective value SCP, and then prove that this bound is tight. For the first 
result, we proceed by taking the optimal stochastic control policy, and construct a feasible solution 
for the DAP by taking expectations over the history. Later, we exploit the concavity of the objective 
and apply Jensen's inequality to show that this new solution attains a greater revenue in the DAP. 



Proposition 4 The optimal objective value of the DAP provides an upper hound on the objective 
value of the optimal policy, i.e. J'^ < J^. 

Now we complete the analysis by lower bounding the yield of the stochastic policy in terms of the 
DAP objective. In proving that bound, we look at N* , the first time that any advertisers contract is 
fulfilled or the point is reached where all arriving impressions need to be assigned to the advertisers. 
We refer to the time after N* as the left-over regime. The first key observation in the proof is that 
before time N* , the controls of the stochastic policy behave exactly as the optimal deterministic 
controls. The second key observation is that the expected number of impressions in the left-over 
regime is 0(\/]V), and the left-over regime has a small impact on the objective. 



Theorem 2 Let 



the expected yield under the stochastic policy fi^ . Then, 



N 



N 



>jl>l 



1 



^here K{p) = ^^Eae^,^- 
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Proof. The first bound follows from Proposition |4j We now prove the second bound. 

Let 5*^ (J = Yl"=i (1 ~ -^ii'^iXQi))) ^i.aiQi) be the total number of impressions assigned to advertiser 
a by time n when following the stochastic policy /i^. Additionally, we denote by = {S^^^^}aeA the 
random vector of impressions assigned to advertisers. Then, Xn,a = Ca — ^ is the total number of 
impressions left to assign to advertiser a to fulfill the contract, and m = N — n is the total number 
of impressions remaining to arrive. 

To simplify the proof, we let Co = N — X]ae.A total number of impressions that are not 

assigned to any advertiser (accepted by AdX and discarded), and we refer to 5^q = n — "^^eA ^n.a as 
total number of impressions not assigned to any advertiser by time n when following the stochastic 
policy fi^ . Because Cq is the total number of impressions we can dispense of, when the point is 
reached that Sn.o = Cq, then all remaining impressions need to be assigned to the advertisers. 

Let the random time N* = inf 1 1 < n < iV : Xn.a = for some a G ^ or Xlae^ ^n.a = ""t-} be the 
first time that any advertiser's contract is fulfilled or the point is reached where all arriving impres- 
sions need to be assigned to the advertisers. Clearly, A'^* is a stopping time with respect to the 
stochastic process {5'^}„=i,...,Ar. 

In the following, let i?^ be the revenue from time n under policy fj,^ . Similarly, we denote by i?„ 
the revenue from time n when the deterministic control are used in an alternate system with no 
capacity constraints. Because the deterministic controls are time-homogeneous, and the underlying 
random variables are i.i.d., then the random variables {Rn}n=i,....N are i.i.d. too. Moreover, it is the 
case that Ei?„ = . Notice that when n < N* , the controls of stochastic policy /i^ behave exactly 
as the optimal deterministic controls. Thus, R„ = R!^ for n < N* . Using this fact together with the 
fact that A'^* is a stopping time we get that 



N 



N* N 
n=l n=N*+l 



>E 



^Rn 



:EN*J[', (8) 



where the inequality follows from the non- negativity of the revenues, and the last equality from 
Wald's equation. Then, we conclude that J§/J^>EN*/N. 

Next, we turn to the problem of lower bounding EA^*. Before proceeding we make some definitions. 
We define by Sn,a the number of impressions assigned to advertiser a by time n when following the 
deterministic controls in the alternate system with no capacity constraints. As for the revenues, it is 
the case that 5„_a = 'S','^,q for n < N* . We define Sn.a in a similar fashion. 

Let Na = inf {n > 1 : Sn.a = C'a} be the time when the contract of advertiser a G ^ is fulfilled, and 
A^o = inf {n > 1 : Sn.o = Co} be the point in time where all arriving impressions need to be assigned to 
the advertisers. Even though these stopping times are defined with respect to the stochastic process 
that follows the deterministic controls, it is the case that A^* = miiiaeAoi^a} ■ In the remainder of 
the proof we study the mean and variance of each stopping time, and then conclude with a bound 
for EA^* based on those central moments. 

For the case of a G the summands of Sn.a are independent Bernoulli random variables with 



success probability pa- The success probability follows from (3a). Hence, Na is a negative binomial 
random variable with Ca successes and success probability pa- The mean and variance are given by 
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¥,Na = N, and Var[A^a] = N^-^, where we used that pa = Ca/N . Similarly, for the case of a = 0, 
now the summands of Sn^ are Bernoulli random variables with success probability po- Hence, A^o is 
a negative binomial random variable with Cq successes and success probability Pq. 

Finally, using the lower bound on the mean of the minimum of a number of random variables of 



Aven ( 1985 ) we get that 



= E min{A^J > min EiV^ - ^ / — ^ V Var[A^J 

aeAo aeAo V ^+1 , 




aeAo 



The result follows from combining ([s]) and □ 

In terms of yield loss, our previous bound can be written as — Jn ^ ^^K{p)J^ ^ achieving 
an 0{^/N) loss w.r.t the optimal online policy. In particular, we may fix the capacity to impression 
ratio of each advertiser, and consider a sequence of problems in which capacity and impressions are 
scaled up proportionally according to p. Then, the yield under policy p^ converges to the yield of 
the optimal online policy as N goes to infinity. 

A key observation in proving the last theorem was that the number of impressions in the left-over 
regime is 0{^/N). In fact, using a Chernoff bound, we may show that the probability that the number 
of impressions in the left-over regime exceeds a fraction of the total impressions decays exponentially 
fast. 

Corollary 1 The probability that the number of impressions in the left-over regime exceeds a fraction 
e> of the total impressions decays exponentially fast, as given by 

F{N -N*>eN}<Y^ exp(-2eVaiV). 

aeAo 

Proof. We prove the complement, that is, the probability that N* > (1 — €)N converges expo- 
nentially fast to one. Notice that N* > (1 — €)N if and only if by time (1 — €)N the contract of 
each advertiser is not yet fulfilled (5'(i_e)7v,a < Ca), and the point where all impressions need to 
be assigned to advertisers has not been reached (5'(i_e)7v,o < Co)- Combining De Morgan's law and 
Boole's inequality we get that 

P{N* > (1 - e)N} = P{5(i_,)A,,, < a Va G A} > 1 - J] > CJ. 

aeAo 

Recall that 5'(i_e)Ar_o is the sum of (1 — e)A'^ independent Bernoulli random variables with success 
probability pa- Hence, we conclude by applying Chernoff 's bound to the each summand to obtain 

nSii-e)N,.>Ca}<exp{-2e'paN). □ 



The policy described in 3.3 is static in the sense that it does not react to changes in supply: 
the dual variables v are computed at the beginning and remain fixed throughout the horizon. To 
address this issue, in practice, one would periodically resolve the deterministic approximation Q. 



Recently, Jasin and Kumar (2010) showed that carefully chosen periodic resolving schemes together 
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with probabilistic allocation controls can achieve bounded yield loss w.r.t. the optimal online policy. 
It is worth noting that those results do not directly apply to our setting: they consider a network RM 
problem with discrete choice, while our model deals with jointly distributed (and possibly continuous) 
placement qualities and AdX. Nevertheless, by periodically resolving the DAP one should be able to 
obtain similar performance guarantees for the yield loss of the control. 

5. Data Model and Estimation 

We have thus far assumed that any user could be potentially assigned to any advertiser. In practice, 
however, advertisers have specific targeting criteria. For instance, a guaranteed contract may demand 
for females with certain age range living in New York, while other contract may demand for males in 
California. In this section we give a parametric model based on our observation of real data, which 
takes into consideration that advertisers demand for particular user types in their contracts. 

Instead of grouping user types according to their attributes, we aggregate user types that match 
the criteria of the same subset of advertisers. This has the advantage of reducing the space of types 
to a function of the number of advertisers (which is typically small in practice) rather then the 
number of possible types (which is potentially large). Hence, a user type is characterized by the 
subset of advertisers T that are interested in it. In the following, we let T be the support of the 
type distribution, and tt{T) the probability of an arriving impression being of type T. As before we 
assume that, across different impressions, types are independent and identically distributed. Given a 
particular type T, the predicted quality perceived by the advertisers within the type is modeled by 
the non-negative random vector Q{T) = {Qa(T)}a(zT- 

Even if the total number of impressions suffices to satisfy the contracts, i.e. '^o.^aP'^ — 
inventory may not be enough to satisfy the contracts targeting criteria. Our algorithm guarantees 
that the total number of impressions Ca is always respected, yet some advertisers may be assigned 
impressions outside of their criteria. If an impression of type T happens to be assigned to an advertiser 
a ^T, the publishers pays a nonnegative goodwill penalty Tq . These penalties allow the publisher 
to prioritize certain reservations, specially when the contracts are not feasible. Thus, the ex-ante 
distribution of quality is given by the mixture of the types distribution with mixing probabilities vr(T). 
Notice that all our previous results hold if we apply the same analysis to the mixture distribution. 

5.1. Estimation 

Although the number of types may be exponential in A, in practice we observe that a linear number 
of them suffice to characterize 98% of the inventory. We observe that the predicted quality perceived 
by the advertisers within a type is approximately log-normal. This can be seen in Figure |3| where 
the empirical distribution of log-quality is graphically represented for a type with two advertisers 
(data is log-transformed). The histograms on the diagonal show the marginal log-quality of each 
advertiser, which approximately resemble a normal curve. On the off-diagonals, scatter plots show 
the correlation between advertisers, which is strongly positive. In some sense this is expected, since 
many advertisers have similar targeting criteria. 
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Figure 3 Graphical representation of the empirical distribution of log-quality for a type with two advertisers (data 
is log-transformed). 



Given a particular type T, we assume that quality follows a multivariate log-normal with mean 
vector ht and covariance matrix for the advertisers in the type, and takes a value of — for 
advertisers not in the type. The total distribution of quality is given by the mixture of these types 
distribution with mixing probabilities vr(T). Thus, we have that 

j InM {fJ.T,^T), foraGT, 
<3~S f w.p. 7r(r). 

To perform the estimation we analyzed data from four different publishers for a consecutive period 
of seven days. First, the capacities of the reservations were used to compute the ratios p. Second, logs 
were analyzed to estimate the types' frequencies, and the parameters of the underlying log- normal 
distributions (using maximum likelihood estimation). 

Bidding data from the same period of time was used to estimate the primitives of the AdX. With 
multiple bidders, AdX runs a sealed bid second-price auction. In this first approach to the problem, 
we assume that bids are independent of the quality of the impressions. We analyze the first and 
second highest bids for the inventory submitted to AdX. We denote by {(-8^' -^2")}™=! m 
sampled highest and second highest bids from the exchange. Sample data is used to compute the two 
primitives of our model: (i) the complement of the quantile of the highest bid p{s) , and (ii) the revenue 
function of r(s). Both functions are estimated on a uniform grid of survival probabilities in 

the [0, 1] range. 

First, for each point in the grid j, the price pj = p{sj) is estimated as the (1 — Sj)-th population 
quantile of the highest bid. Then, using sampled bids, we estimate the revenue function w.r.t. to 
prices at the grid points as 

1 ^' 

<P^^ = M E HBT>p,}max{B^,p,} (10) 

m—l 



Finally, the revenue function is obtained by composing (10) and p{s). AdX data is available only for 
the first two publishers. 
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rpypgg Survival Pnobahility (s) 

(a) User type-advertiser graph. (b) Estimated survival probability and revenue func- 

tion for AdX. 



Type 


Ads 


vr(T) 




XI 2^ 


Ti 


{1,2,3} 


0.2 


/ 7.8155 \ 

7.8155 
V 7.8155/ 


/0.3 0.1 0.1\ 

0.1 0.3 0.1 
Vo.l 0.3 0.1/ 


T2 


{1,2} 


0.3 


/- 6.6755 \ 
V 7.0655 ) 


( 0.3180 0.1649 N 
I 0.1649 0.3602 ) 




{2,3} 


0.1 


/ 6.6355 \ 
V 7.8055 ) 


('0.4347 0.2357 N 
I 0.2357 0.4367/ 




{1,3} 


0.4 


( 7.2155 \ 
V 6.9155 ) 


( 0.23 0.05 ^ 
I 0.05 0.40 ) 



(c) Parameters of the distribution of log-quality. 
Figure 4 Description of Instance 1. 



Figure 4a describes Instance 1, a publisher with 4 types and 3 advertisers. The estimated survival 



probability and revenue function for the publisher is shown in Figure 4b The parameters for the 
remaining publishers are available at the webpage of the first author. 



6. Experimental Results 

Two experiments were conducted to study our algorithm. First we study the impact of introducing 
an AdX on the publisher's yield. Second, we compare the previously known primal-dual approach to 
ad allocation that is non-parametric to our approach here which is parametric. 

6.1. Impact of AdX 

This first experiment explores the potential benefits of introducing an AdX, and how the publisher 
can take advantage of it. We study the impact of the trade-off parameter 7 on both objectives, that 
is, the quality of the impressions assigned to the advertisers, and the revenue from AdX. The limiting 
choices of 7 = 0, and 7 = c« are of particular interest. The first choice represents the case where 
the publisher disregards the quality of the impressions assigned to the advertisers, and strives to 
maximize the revenue extracted from AdX. Here the publisher strategically picks the reserve price so 
that just enough impressions are rejected to satisfy the contracts. In the second choice, the publishers 
prioritizes the quality of the impressions assigned, and submits the remanent inventory to AdX. We 
use this the baseline to which we compare our method. 
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450 



plots, 



plots the quality and revenue relative to the case 7 = oo, as a function of 7. Figure 
a quality vs. revenue graph, the objective values of the optimal solutions for the different choices of 7, 
together with the Pareto frontier. Both plots are for Instance 2, given in !S 



Instance 1 



7 





0.001 


0.01 


0.05 


0.075 


0.1 


Yield 


110.94 


112.78 


128.81 


202.46 


249.11 


296.95 


Quality 


1107.32 


1779.07 


1801.00 


1864.95 


1891.19 


1913.78 


Revenue 


110.94 


111.00 


110.80 


109.21 


107.27 


105.57 


7 


0.25 


0.5 


0.75 


1 


10 


oo 


Yield 


590.54 


1098.43 


1608.80 


2122.99 


20764.82 


oo 


Quality 


1998.31 


2044.89 


2055.72 


2061.33 


2072.22 


2075.52 


Revenue 


90.97 


75.98 


67.00 


61.66 


42.61 


38.48 



Instance 2 



7 





0.001 


0.01 


0.05 


0.075 


0.1 


Yield 
Quality 
Revenue 


428.73 
483.02 
428.73 


429.02 
545.27 
428.47 


434.22 
573.23 
428.49 


459.57 
676.34 
425.75 


477.39 
720.31 
423.37 


495.66 
752.35 
420.42 


7 


0.25 


0.5 


0.75 


1 


10 


oo 


Yield 
Quality 
Revenue 


617.04 
843.47 
406.17 


834.39 
880.69 
394.05 


1056.05 
891.49 
387.43 


1279.88 
896.89 
382.99 


9425.63 

906.46 

360.99 


oo 
907.05 
356.11 



Table 2 Expected yield, advertisers' quality and revenue from AdX for two instances, and different choices of 7. 



The experiment was conducted as follows. First, we set up a grid on the trade-off parameter 7. 
Then, we solve the publisher's problem as given in Q. The resulting policies are evaluated using a 
fluid limit (see EC.4). Table [2] reports the expected quality and revenue for different choices of 7. 



Figure 5a plots the quality, and revenue relative to the baseline function of 7. In Figure [5b| 



we plot, in a quality vs. revenue graph, the objective values of the optimal solutions for the different 
choices of 7, together with the Pareto frontier. 
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Discussion. Results confirm that, as we increase the trade-off parameter 7, the quality of the 
impressions assigned to the advertisers increases, while the revenue from AdX subsides. Interestingly, 
starting from the baseline case that disregards AdX (7 = 00), we observe that the revenue from AdX 
can be substantially increased by sacrificing a small fraction of the overall quality of the impressions 
assigned. For instance, by exploiting strategically the AdX, the publisher can increase AdX's revenue 
by 8% by giving up only 1% quality. Conversely, starting from the case that disregards the advertiser's 
quality (7 = 0), the publisher can raise the quality in a large amount at the expense of a small 
decrease in AdX's revenue. 

Alternatively, the previous analysis can be understood in terms of the Pareto frontier. Results show 
that the Pareto frontier is highly concave, relatively horizontally flat around 7 = 00, and vertically 
flat around 7 = 0. This explains the huge marginal improvements at the extremes. There are several 
advantages to the quality vs. revenue representation. First, the Pareto frontier allows for quick grasp 
of the nature of the operation. When the publisher's current operation is sub-optimal, its performance 
point should lie in the interior of the frontier. In this case, the Pareto frontier allows the publisher to 
measure its efficiency, and quantify the potential benefits an optimal policy may introduce. Second, 
when the choice of the trade-off parameter is not clear, the publisher may impose a lower bound 
on the overall quality of the impressions, and instead maximize the total revenue from AdX. The 
efficient frontier provides the maximum attainable revenue, and the proper 7 to achieve the quality 
constraint. 



6.2. Comparison with the Primal-Dual Approach 

In this second experiment we study the performance the our algorithm, and contrast it with a Primal- 



Dual (PD) method. This experiment is discussed in detail in ^EC.l , but we give the full experiments 
and discussion here for the benefit of the reader. Since no existing PD method is known yet for the 



AdX problem, we consider instead the case with no AdX. The Primal-Dual approach (Devenur and 



Hayes||2009 ) , uses a sample from data to estimate the dual variables and uses it in a bid-price control 



policy. In contrast, our algorithm, as stated, assumes the parameters of the quality distribution are 
known, and uses that to estimate the dual variables. So we do not need to use a sample. Of course 
in practice, the parameters need to be learned, and so we would need to use a sample of the data in 
order to learn them; but in many settings (including online advertising) it is reasonable to assume 
that we at least know the form of the distribution (e.g., normal, exponential, Zipf), albeit not the 
specific parameters (mean, variance, covariance, etc.). The techniques in Devenur and Hayes (2009) 
are powerful because they don't need to assume anything about the distribution, but it is important 
to ask what can be gained from knowing the form of the distribution, which is what we do in the 
remainder of this section. 

In order to objectively assess the performance of our algorithm we adopt the user type model 
described in ^ as a generative model. The generative model is used to generate sample data on which 
both our algorithm and a PD method are tested. The advantages of adopting a generative model are 
twofold. First, it allows us to compute the truly optimal policy fi'^^'^. Second, the true performance 



of any policy can be evaluated efficiently using a fluid limit (see Section EC. 4). 
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Figure 6 



10000 20000 30000 40000 50000 10000 20000 30000 40000 50000 

Training Set Size Training Set Size 

(a) Average Yield (b) Std. Dev. of Yield 

Average (a) and standard deviation (b) of yield as a function of training set sample size; results are shown 
for the parametric method (EST) based on our policy fi and the primal-dual method (PD) as in Deve nur| 



and Hayes, (2009) . Both policies converge to the optimal yield, but EST converges faster, and with less 



variance. 



The computational experiment is conducted as follows. First, a training data set of M impressions 
is generated. We denote the sampled quality vectors by {qm}m=i- Then, we estimate the parameters 
of the model on the training set as follows. For each type we estimate the type probabilities ttt; and 
mean /i^, and covariance matrix Tjt of the logarithm of the qualities. Next, the dual problem Q is 
solved on the estimated parametric model using a Gradient Descent Method as described in §EC.3[ 
Note that, since no AdX is considered, the maximum expected revenue function is the identity. 
Using the optimal solution v^^^ we construct a policy, which be refer as /i^^"^. 

Simultaneously, we employ the PD method on the training data. The PD method amounts to 
solving a sample average approximation of problem Q , which results in the following linear program 



Taim^'^X„, + ^PaVa (11) 



M 

T M ^ 

rn—l 

S.t. Xm+Va> qni.A, Vm, O 

A„, > Vm. 



The linear program is solved using CPLEX 12. Again, using the dual optimal solution we 
construct a policy jjF^. 

Afterwards, we assess the performance of both policies using a fluid limit. These steps are replicated 
on 50 different training sets. Table [3] reports the average results over the training sets for different 
sizes of training sets, and instances. Plots of the results for a given instance are shown in Figure [6] 

Discussion. Results show that for both algorithms, as the size of the training set increases, the 
optimality gap decreases at a rate of 0{M^). However, the parametric method performs uniformly 
better that the non-parametric PD method. Additionally, the variability across different training sets 
diminishes as the size of the training set increases. Indeed, we observe that the standard deviation 
over training sets converges to zero for both methods, but the convergence is faster for the parametric 
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Instance 1 (A = 3, T = 4, OPT = 2075.09) 



Training 


EST 




PD 




Set Size 


mean 


std.dev. 


mean. 


std.dev 


100 


2004.16(3.42%) 


33.978 


1990.32 (4.08%) 


37.552 


1000 


2053.41 (1.04%) 


10.008 


2047.92(1.31%) 


12.365 


2500 


2065.12 (0.48%) 


4.956 


2062.76 (0.59%) 


5.838 


5000 


2068.44 (0.32%) 


3.681 


2066.99 (0.39%) 


4.224 



Instance 2 (A = 6, T = 10, OPT = 907.44) 



Trainina 


EST 




PD 




Set Size 


mean 


std.dev. 


mean. 


Std.dev 


1000 


889.58 (1.97%) 


8.861 


882.77 (2.72%) 


12.829 


2500 


894.43(1.43%) 


7.485 


887.64 (2.18%) 


10.418 


5000 


898.59 (0.98%) 


5.231 


892.51 (1.65%) 


7.625 


10000 


901.13(0.70%) 


3.588 


897.42(1.10%) 


4.692 


25000 


904.69 (0.30%) 


1.712 


901.97 (0.60%) 


2.720 


50000 


905.03 (0.27%) 


1.267 


903.44 (0.44%) 


1.567 




Instance 3 (A = 


17,T=15, OPT = 894.82) 




Training 


EST 




PD 




Set Size 


mean 


Std.dev. 


mean. 


Std.dev 


2500 


859.83 (3.91%) 


9.937 


849.44 (5.07%) 


14.615 


5000 


868.61 (2.93%) 


5.870 


861.06 (3.77%) 


7.954 


10000 


877.59(1.92%) 


5.226 


873.46 (2.39%) 


6.577 


25000 


884.04(1.20%) 


2.585 


881.13(1.53%) 


3.747 


50000 


887.34 (0.84%) 


1.926 


885.11 (1.08%) 


2.728 



Instance 4 (A = 14, T = 10, OPT = 928.76) 



Training 


EST 




PD 




Set Size 


mean 


std.dev. 


mean. 


std.dev 


2500 


892.55 (3.90%) 


12.886 


888.88 (4.29%) 


13.427 


5000 


903.04 (2.77%) 


8.537 


901.79 (2.90%) 


10.277 


10000 


911.25 (1.88%) 


6.951 


909.96 (2.02%) 


6.935 


25000 


917.30 (1.23%) 


3.353 


915.81 (1.39%) 


3.705 


50000 


921.36 (0.80%) 


2.668 


920.11 (0.93%) 


2.716 



Table 3 Experimental results comparing tiie performance of our parametric method (EST) witli the 
non-parametric primal-dual method (PD). No AdX present in this experiment. 



one. In some sense this is expected, since the true data model follows exactly the distributional 
assumptions. However, the PD method is expected to be more robust to model misspecification. 

Another experiment, though results are not reported, was conducted to test the strength of the 
parametric method on real data. We observed that, when the training set is small (around thousands), 
the parametric method performs better than the non-parametric one. However, as the sample size 
increases the non-parametric method outperforms the other. The rationale for this behavior is that, 
when data is scarce, the parametric method can exploit the distributional assumptions to reconstruct 
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a fair representation of the data. However when the training set is larger, the fit of our model to real 
data is not perfect, and the non-parametric method can withstand deviations more robustly. 

7. Extensions 

In this section we consider a number of extensions of the model and policy from the previous section. 
7.1. Target Quality Constraints 

In section |2.1| we discussed an alternate formulation in which the publisher imposes a minimum 
overall quality for the impressions assigned to the advertisers. This might be more natural for some 
publishers; they might feel more comfortable specifying target quality constraint than picking a 
Lagrange multiplier to weight the impact of quality in the objective. Additionally, in some settings 
the advertisers themselves might demand that certain level of quality is guaranteed. 

In the following, we impose that the average quality of the impressions assigned to advertiser a is 
larger or equal than a threshold value ia- Now, the publisher would strive to maximize the revenue 
from AdX, while complying with the target quality constraints, and the contractual obligations. The 
one-impression DAP would be similar, except that the objective only accounts for AdX's revenue, 
and the inclusion of the constraints 

^[ia{Q)Qa]>L, VoGA (12) 

We attack the problem, as done before, by considering its dual. Let 7a > be the Lagrange 
multiplier associated to ( 12 ). As a side note, problem (|3| can be interpreted as the Lagrange relaxation 



of our new problem w.r.t. the target quality constraints, and the dual variables 7 as the shadow price 
of the target quality constraints. The new constraints preserve the convexity of the program, and 
strong duality still holds. Following the same steps, we obtain the new dual problem 



min I ER ( max{jaQa - ^a} ) + VaPa - laL \ 



which still is a convex minimization problem. The publisher might now jointly optimize over v, and 
7 to construct a provably good policy. Additionally, in a similar fashion to Proposition [6j we may 
compute the directional derivative of the objective w.r.t. the dual variables 7. 

Regarding the performance the bid-price control p^ , Theorem [2] still holds, and the policy asymp- 
totically attains the optimal revenue from AdX, while complying with the delivery targets. However, 
we still need to argue about the expected average quality assigned to the advertisers. Unfortunately, 



for those advertisers whose constraint (12) is binding, our algorithm might not attain the desired 
quality target. Nevertheless, from our asymptotic analysis we may show that the expected average 
quality is lower bounded by 

N 



E 



n=l 



>^E[z:{Q)Q^]>(l--^K{p)]L 



Hence, for advertisers with binding constraint (12), albeit not feasible, the expected average quality 



becomes arbitrary close to the threshold value as the number of impressions in the horizon increases. 
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On the other hand, for the remaining advertisers whose target quahty constraint is not binding, the 
expected average quahty wih surpass the threshold for suitably large A'^. 

7.2. AdX with Multiple Bidders 

Here we generalize our results to the case where multiple buyers participate in the Ad Exchange. 
We model AdX as an auction with K risk neutral buyers. The publisher believes that individual 
valuations are drawn independently from the same distribution with c.d.f -F(-), density /(•), and 
support [po>Poo]- Moreover, we assume that the distribution of the values have increasing failure rates, 
are absolutely continuous and strictly monotonic. The publisher must choose the reservation price p 
that maximizes her expected revenue given that her value for the impression is c> 0. As before, we 
denote by R{c) the optimal expected revenue of the publisher. 



Myerson ( 1981 ) argued that under our assumptions the optimal mechanism is a Vickrey or second- 
price sealed-bid auction. Moreover, it is known that in such auctions bidding the true valuation is a 
dominant strategy for the buyers, and that the optimal reservation price p* (c) is independent of the 



number of buyers (Laffont and Maskin 1980). 

Let Bi,K and B2:k be the order statistics which denote the highest and the second highest bid 
respectively. Given a reserve price p, the item is sold if Bi-k i.e., there is some bid higher than 
the reserve price. The winning buyer pays the second highest bid, or alternatively max{i?2:if,p}, 
since the seller should receive at least the reserve price p. Therefore, the publisher's maximization 
problem is 

R{c) = maxE [1{Bi.,k > p} maii{B2:K,p} + < p}c] . 

p>0 



Notice that the setup of Section 2.2 can be consider as a particular case of a second-price auction in 
which we have only one bidder and i?2:if = 0. 

Recall that, instead of reserve prices, we casted our problem in terms of survival or winning 
probabilities. Then, letting s be the probability than the impression is sold, we have that s = ¥{Bi.k > 
p} = 1 — F^{p) since valuations are i.i.d. Conversely, the reserve price as a function of the survival 
probability is given by p[s) = F^^{1 — (1 — s)^/^), which is well-defined due to the strict monotonicity 
of the c.d.f. In terms of survival probabilities, the problem is now 

R(c) = max r(s) -|- (1 — s)c, 

0<s<l 

where we defined the revenue function as r(s) = r(p{s)), and r(p) = E [l{Bi.ji > p} max{i?2:K,p}]- 

The next proposition shows that the revenue function is regular, and as a consequence all previous 
results hold for the case with multiple bidders. 

Proposition 5 Under the previous assumption, the revenue function r{s) is regular. Moreover, the 
optimal reserve price p* (c) solves 
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when cG\po — ^ / f (po) i Poo] ■ When the opportunity cost is higher than the null price {Opoo), the 
publisher bypasses the exchange (p*(c) =Poo)- Finally, when the opportunity cost is low enough (c < 
Po — 1/ /{pq)), the impression is kept by the highest bidder (p*(c) =Po)- 



Proof. The joint distribution of Bi,k and B2:k has a density function ( Laffont and Maskin||1980 ) 

^K{K - l)F(62)^-V(&i)/(fc2) if 61 > 62 



f{bi,b,) 
Then, we have that 



otherwise 



r{p) = E [1{B2:K > P}B2..K + P1{B,.,K > P, B2:K < p}] 

= / b^fib.M) db2db^+p f{b„b2)db2db^ 

J p J p J p J 

= K{K-1) / b2F{b2y'-^f{b2){l-F{b2)) db2 + KpF{pf-'{l-F{p)) 
Jp 

Continuity of r{s) follows because the p.d.f. is continuous, and p{s) is continuous (if F not strictly 
monotone, the inverse may have jumps). Additionally, we may bound the revenue by 

r{p) < E [l{Bi.,K > p}Bi.,k] < KE [1{B > p}B] < KEB < 00, 

the first inequality follows because Bi-^ is the maximum, the second because any order statistic 
is upper bounded by the sum of the bids, and the fourth because bids are integrable. Moreover, 
integrability of B implies that limp_i.oo 'r'{p) = 0. 

Next, we turn to the concavity of r(s). Differentiating w.r.t to p we get 

-=KF{pr-\F{p)-pf{p)). 

Then, using the fact that ^ = —KF{pY^^/f{j)) we get from the composition rule that 



dr dr 
ds dp 



dp^ f . 1 

P^'^ hip{s)y 



where h{p) = f{p)/F{p) is the hazard rate of the bidder's valuation. Because p{s) is non-increasing 
in s and the h{p) is non- decreasing in p, we conclude that ^ is non-increasing. Thus, the revenue 
function is concave. 

Finally, notice that the that derivative of the objective w.r.t to s is 

P{s)- , ] -c, (13) 
h{p{s)) 

which is non-increasing. When c > Poo we have that (flSj) is negative, so s*{c) = and p*{c) =Pao- 



Similarly, when c<Pq — l/h{po) we that (13) is positive, so s*{c) = 1 and p*{c) =Po- □ 
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7.3. AdX with User Information 

In most systems, the pubHsher shares some user information with the excliange. In turn, the exchange 
may partially disclose the user information to their advertisers. The advertisers may react to this 
information, and bid strategically ( Muthukrishnan||2009 > . In this section we extend our model to the 



case when the bids from AdX are correlated with the quality of the impression (a surrogate for user 
information). For simplicity we consider the case of one bidder. Nevertheless, our analysis can be 
easily extended to the general case. 

Let F{p\Q) = F{B >p\Q} be the conditional probability that the bid from AdX is greater than 
p given that the impression quality vector is Q. Additionally, we define the conditional revenue 
function as r{s\Q) = sF^^{s\Q). The publisher can exploit the correlation between user information 
and bids to update his prior on AdX bids. Conditioning on the impression quality, we obtain that 
the maximum expected revenue under opportunity cost c, denoted by R{c\Q), is now 

R(c\Q) = max r(s\Q) + (1 - s)c. (14) 

0<s<l 

In order to apply the results from the previous sections we require that the conditional revenue 
function r{-\Q) is regular for all qualities Q almost surely. 

The rest of the analysis follows in a straightforward way by conditioning on the impression quality. 
The dual problem Q now reads 



mm i^R ^max{Qa -Va} \Q^ + VaPa^ 



where we replaced the maximum expected revenue by R{-\Q). It is worth noting that now the optimal 
reserve price to be submitted to AdX depends both on the maximum contract adjusted quality, and 
the actual realization of the quality vector. 

8. Conclusion 

Ad Exchanges are an emerging market for the real-time sale of online ad slots on the Internet. 
Despite the popularity of this emerging market, many publishers are currently not jointly optimizing 
their inventory over AdX and their traditional reservations. Instead they first aim to fulfill their 
reservations and then submit their remnant inventory to the exchange. In this work, we show that 
there are considerable advantages for the publishers from jointly optimization over both channels. 
Publishers may increase their revenue streams without giving away the quality of service of their 
reservations contracts, which still represents a significant portion of their advertising yield. Our 
approach helps publishers determine when and how to access AdX to complement their contract sales 
of impressions. In particular, we model the publishers problem as a stochastic control program and 
derive an asymptotically optimal policy with a simple structure: a bid-price control extended with a 
pricing function for the exchange. We also hope our insights here will help understand ad allocation 
problems more deeply. 

Internet advertising, and in particular AdX, is likely to prove to be a fertile area of research. 
There are several promising directions of research stemming from this work. One intuitive approach 
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to improve the performance of a control, which is appeahng for its simphcity, consist on resolving 
the deterministic approximation periodically throughout the horizon. In a follow-up work we intend 
to show that one can indeed improve on the static control and obtain sharper bounds by resolving 
the DAP. Another problem that needs further study is that of learning in the case of unknown 
distributions, which is of great importance given the fast-paced and changing nature of the Internet. 
There exists independent research on online algorithms for capacity allocation and online pricing for 
repeated auctions, but none on the joint optimization problem. Finally, as more publishers reach 
out for AdX, advertisers will have the opportunity to buy their inventory from either market. The 
existence of two competing channels, the exchange as a spot market and the reservations as future 
market, introduces several interesting research questions. For example, how should publishers price 
their contracts and allocate their inventory, and how should advertisers hedge their campaign between 
these two markets. We hope that this work pave the way for further research on this important topic. 
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Appendix A: Proofs of Statements 

Proof of Proposition^ First, observe that for all c the objective function of ([T]) is concave and continuous 
in s, and the feasible set is compact. Hence, by Weierstrass Theorem the set of optimal solutions is non-empty 
and compact. Thus, both R{c) and s*(c) are well-defined. 

Second, R{c) > c follows from letting s = 0. To see that i?(c) is non-increasing, let c < c', and s* be the optimal 
solution under cost c. Then, R{c) = r{s*) -I- (1 — s*)c < r{s*) -I- (1 — s*)c' < R{c') where the first inequality 
follows because s* < 1, and the second because no solution is better than the optimal. To see that R{c) — c is 
non-increasing, let c' < c, and s* be the optimal solution under cost c. Then, by a similar argument we get that 
R{c') — c' > r{s*) — s*c' > r(s*) — s*c — R{c) ~ c. Convexity follows in a similar way (this is a standard result). 

Third, observe that the objective function of ([T]) is jointly continuous in s and c. Thus, by the Maximum 
Theorem i?(c) is continuous in c, and s*(c) is upper-hemicontinuous. 

Finally, because r{s) -|- (1 — s)c has decreasing differences in (s, c) and the feasible set is a lattice, by Topkis's 
Theorem s*(c) is non-increasing in c. The result for p*{c) follows from the fact that F~^{s) is non-increasing 
in s. □ 

Proof of Theorem [7j The optimality conditions of v for problem Q imply that the directional derivative 
of along any direction is greater or equal to zero. In particular for each advertiser a£ A it should the case 
that Vi^'ip{v) > 0, and V-i^-i/'l'^) ^ 0- Applying proposition |6] to both directions, together with the fact that 
there is zero probability of a tie occurring, we get that 

E [(1 - S* (Q„ - Va)) l{Qa -Va> Q a' " ^a' Vo' G Aq}] = Pa, 

and the result follows. □ 

Proof of Proposition^ Let s — {sn{-)}n^i,...,N and r= {jn(-)}n=i,...,jv be any feasible vectors of controls. 
Let s be the mean of the controls (in terms of prices p would be the generalized F-mean), which is defined 
point-wise s—j^ '^n=i ^n- Similarly, let i be such that ia = ^ '^n=i point- wise for all a€ A. We will show 
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that solution in which (s, i) are used for all impressions is a feasible control with greater or equal revenue than 
the original one. 

First, for the feasibility of (s, z) observe that for each advertiser aG A 

N 



Q 



= NEq [z,] 



where the first equation follows from the feasibility of {p, i) and the linearity of expectation, and the second 
from substituting la pointwise for Q. Clearly, from the convexity of V it follows that (s,T) e V. 

Second, we denote by J'^{s,i) and J^{s,i) the objective value of the solutions (s, z) and (s, z) respectively. 
We have that 

JV iV 



j^(s,r) = E 



<E 



Nr 



\ n — 1 / a^A n = l 



:NE 



• (s) + E 



where the inequality follows from the concavity of the revenue function, and the second equality from substi- 
tuting s and I pointwise for all Q. □ 

Given a subset of the quality space Z? C f2, we define the measure Vr{D) as the probability that the quality 
vector belongs to that subset and the impression is rejected by the Ad Exchange when the optimal survival 
probability is used. More formally, 



Vr{D)=E 



1 - s* ( max{Q, - vS] ] HQ e D] 



Notice that the latter is not a probability measure since Pfl(ri) < 1. Proposition [6] characterizes the directional 
derivative of the objective function of the dual along some directions that, as we will show later, are of particular 
interest. Results are given in terms of the measure P^. 

Proposition 6 Given a subset a ^ A, the directional derivative of the objective function of the dual w.r.t. 
directions 1^ and — 1^ are respectively 

Vi^Vl^) = -Pfl \ max{Q„ - > max {Q^ - ^ + V'pa, 

I a£a a£AQ\a I ^ — ^ 

V_i„V(«) =IPb \ max{(3„ > max {Q^ - ^ - V'pa- 

Proof of Proposition^ We consider first the direction 1^,. Notice that the random function 
R(maiCasAo{Qa — Va}) Is convex, and thus directionally differentiable. We first show that ip{v) is finite. From 
Assumption [T] we have that the revenue function is bounded by r(s) < M, and thus R{c) < M + max(c, 0) < 
M-|- |c|. Therefore, using the triangle inequality we obtain that 

0(t;)<M + E|max{Q„-t;J| <Af + y E|g,| + |w,| <oo 



aeA 



We can now apply Theorem 7.46 in Shapiro et al. (2009) and obtain that ij{v) is directionally differentiable at 
V and that one can exchange expectation and directional derivative. Putting all together we get that 



Vi„'(/'(w) =E 
= E 



Vi^i? ( max{Qa - Va} 

\aeAo 



Ka^AQ 



R' ( max{(5a - Va} ) Vi^ <^ \TlBx{Qa - Va} 



a^Ao 
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where the second equation follows from the chain rule. We conclude by the fact that R'(c) — 
1 - s*(c) and Vi„ {max„g^Q{(5„ - w^}} = -1 {maXa^^lQa - v^} > nia:>CaeAo\c.{Qa ~ Va}}. A simi- 
lar result follows for the opposite direction — 1^ from the fact that V_i^ {niaXag^ulQa ~ '^a}} — 
1 {majiafZc^lQa - Va} > max^g^g \„ { - □ 

Proof of Proposition^ The proof proceeds by contradiction, that is, we assume that there is no feasible 
flow. First, we cast the feasible flow problem as a maximum flow problem. Feasibility would imply the existence 
of a flow with value 1 — P(0-tie). But since we assume that no such feasible flow exists, by the max- flow min-cut 
theorem there should exists a cut with value strictly less than 1 — P(0-tie). The contradiction arises because the 
optimality conditions of v for the dual problem Q imply that the every cut is lower bounded by 1 — P(0-tie). 

In order to write the feasible flow problem as a maximum flow problem, we first add a source s and a sink t. 
Second, we add one arc from s to each node associated to a non-empty subset S C (left-hand side nodes) 
with capacity P(5-tie). Third, we add one arc from each advertiser a € Aq (right-hand side nodes) to t with 
capacity pa- Lastly, we set the capacity of arcs from S' to a e S* to infinity. 

Now, since no feasible flow exists, by the max-flow min-cut theorem there should be a cut with value strictly 
less than 1 — P(0-tie) . Let aC Aq be the advertiser nodes (right-hand) belonging to the t side of a minimum 
cut. Figure [t] shows the minimum cut. Next we argue that subset nodes in the s side verify that S (la — 0, 
while those in the t side verify that 5 Ha 7^ 0. First, because the cut has minimum value, there is no arc from a 
subset node to an advertiser node crossing the cut (those arcs have infinity capacity). Equivalently, within the 
s side of the cut, all subsets nodes S C Aq should verify that SOa — ^d. Second, observe that any subset node 
with 5 n a = in the t side of the cut could be moved to the s side of the cut without increasing the value of 
the cut. Hence, with no loss of generality we can assume that all subset nodes in the t side of the cut verify 
that 5*0 a 9^0. 

As a consequence, the only arcs crossing the cut are those from the source to the subsets S'n a ^ 0, and 
those from advertisers Aq \ a to the sink. The value of this cut is 

Because the value is strictly less than 1 — P(0-tie) we get that 

J2 PiS-tie)<J2Pa, (15) 

where we used that J2aeA P<^ + Po^ ]P(0-tie). 

Next, we look at the optimality conditions of v for the dual problem Q. We distinguish between the case 
that ^ a and € a. First suppose that ^ a, and consider the direction — 1„ that has a —1 if a S a and 
elsewhere. According to proposition |6] the directional derivative of the objective at v is 

V_i„V(«) =]Pii \ inax{(5„ > max {Q^ \ -^^Pa 

= m-tie)-$]pa, 

where we have written the event that the maximum is verified non-exclusively by some advertiser a € a as all 
S-ties in which some advertiser a € a is involved. The optimality of v implies that the directional derivative 



along that direction is greater or equal to zero, contradicting equation ( 15 1 



When € a we consider the direction that has a 1 if a ^ a and elsewhere. The direction derivative is 



now 



^ ^a\J''<"^^~'^r \ max{Q^-w„}> max {Qa-v^}\^ 

■^^ I a(^A\a aeQU{0} I — ' 



a^A\oi 



- lP(5-tie)-|- P^^ E P(^-tic)-^P., 

S'lAa:S'^A\a a^A\a: SQAa:Sr\a:i^<i aea 
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Subset nodes Advertiser nodes 

Figure 7 The flow problem In the bipartite graph with a minimum cut. A source s connected to the subset nodes 
and a sink t connected to the advertisers nodes was Included, a C Ao Is the subset of advertiser nodes 
(right-hand) belonging to the t side. Note that no there Is no arc from a subset node to an advertiser 
node crossing the cut. 



where in the second equation we have written the event that the maximum is verified exclusively by some 
advertiser a € a as all S'-ties in which only advertisers in a are involved. Again, the optimality of v implies 
that the directional derivative along that direction is greater or equal to zero, contradicting equation (15). □ 



Proof of Proposition^ Let ^* be the optimal policy for the stochastic control problem. Let s — 
{sn{-)}n=i,...,N and « = {«„(-)}„=i,...,]v be deterministic vectors of controls defined as 

s„ (Q) = [<* (Q) I Q] VQ pointwise, 

i„,a(Q)-E^„ IQ] VQpointwise,a€ A 

where the expectation is taken over the history of the system until n, which is denoted by and conditional 
on a particular realization of Q. The resulting controls are independent of the history, and dependent only on 
the realization of Q and the impression number n. Thus, they fulfills the first approximation and they are valid 
deterministic vectors of controls. We will show that (s, i) is feasible for the DAP, and that its objective value (in 
the DAP) dominates the optimal objective value of the SCP. Then, we may conclude that < J^{s,i) < J^, 
because no feasible solution is better than the optimal. 

First, for the contract fulfillment constraint we have that for each advertiser aG A 

N 



^(i-x„«-(gj))/^;(g„) 



n—1 n—1 

where the first equality follows from taking expectations to the almost sure contract fulfillment constraint of 
/i*, the second from the tower rule, and the third from substituting s and i pointwise for all Q and the fact 
that impressions are i.i.d. Non-negativity of the controls follows trivially. Additionally, is it not hard to show 



that V 



g_4 ■'n,a\ 



+ < 1 for all n. Thus, (s,J) is a feasible deterministic control. 



Second, the objective value of the optimal stochastic control is bounded by 



-E 



^E^„ [r «*(QJ) I QJ +5]Q„,.E^„ I Qn] 



<E 



(Sn(Qn)) + ^ Qn,Jn,a{Q) 
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where the first equahty foUows from the tower rule and because Q„ is measurable w.r.t. the conditional 
expectation, and the inequality from applying Jensen's inequality to the concave revenue function. □ 

Proof of Proposition^ Because 5 = 0, then it is not hard to show that F~^{s) ~ 0, and that the revenue 
function is r(s) =0. Hence, the revenue function is regular and satisfies Assumption [l] Moreover, the optimal 
survival probability is s*(c) — 0, and R{c) = c. The result follows from substituting these functions in Theorem 

m □ 
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Appendix EC.l: Comparison to the Primal-Dual Method 

Consider the allocation problem faced by a publisher in display advertising in which arriving impres- 
sions need to be assigned to advertisers, and there is no option of sending to an exchange. This 
problem is a particular case of our model where the winning bid random variable is identically 
zero, i.e. B = 0. The following proposition shows that the optimal controls admit simple analytical 
expressions. 

Proposition 7 Suppose that ties have zero probability. Then, in the case without AdX the optimal 
controls are Ia{Q) = 1 {Qa — Va> Qa' — Va' Va' e ^o} where v = {va}aeAo satisfies f o = and 

The resulting decision rule argmaX(j{Qa — Va} is identical to the rule studied in previous work 



(e.g., Devenur and Hayes (2009)), where Va is an optimal dual variable resulting from solving an 
assignment problem on a sample of the data, where the distribution is unknown. Roughly speaking, 
in Devenur and Hayes (2009) (and similarly in other work Feldman et al. (2010), Vee et al. (2010), 



Agrawal et al. (2009)), it is shown that as long as the sample is of size ~ en, the overall assignment 



will be ~ e close to the optimal offline solution. 

In our model, the parameters of the quality distribution are known, so we do not need to use a 
sample. Of course in practice, the parameters need to be learned, and so we would need to use a 
sample of the data in order to learn them; but in many settings (including online advertising) it is 
reasonable to assume that we at least know the form of the distribution (e.g., normal, exponential. 



Zipf), albeit not the specific parameters (mean, variance, covariance, etc.). The techniques in Devenur 



and Hayes (2009) are powerful because they don't need to assume anything about the distribution. 



but it is important to ask what can be gained from knowing the form of the distribution, which is 
what we do in the remainder of this section, both analytically and experimentally. 

More formally, suppose the distribution of quality is not known with certainty, but we have at our 
disposal a sample of M quality vectors {(?m}^=i that may be used to pin-down the distribution. Addi- 
tionally, it is known that the qualities are drawn independently from a population with continuous 
density g{x\9), where is an unknown parameter to be estimated. Let G{x\9) be the c.d.f. which we 
assume to be strictly monotonic. For simplicity, we deal with the case of one advertiser with capacity 
to impression ratio of p. From Proposition [7] the optimal DAP control is the (1 — /3)-quantile of Q, 
that is, V = G^^{p\6). We compare the asymptotic efficiency of a parametric and a non-parametric 
estimation of the model. 

Parametric estimation method. Let 0™'° be the maximum likelihood estimator (MLE) of the 
unknown parameter 6. That is, 6™^'' is an optimal solution of the program max^ ^^^-^ log (/((7„i|0). 
Once we have our estimator, we plug-in the estimated distribution in the dual problem, and solve 
for the optimal dual variable Again, from Proposition [t] we have that the optimal dual variable, 
given our maximum likelihood estimation, is given by = {p\6^]^l°) . 

In turn, by the invariance property of the MLE, it is the case that v"^^ is the maximum likeli- 



hood estimator of the true optimally dual variable v (see, e.g., Casella and Berger (2002)). As a 
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consequence, under some regularity conditions, we have that our new estimator is consistent, asymp- 
totically efficient, and asymptotically normal 

where the u{9) is the Cramer-Rao lower bound on the variance of any unbiased estimator. The 
Cramer-Rao lower bound is u(^) = (^(pI^)) I{6y\ where I{e) =E In g{Q\9)Y is the Fisher 
information of parameter 9. 

Non-parametric estimation method. Considering the Sample Average Approximation (SAA) of the 
dual problem Q we can obtain a non-parametric estimator of the truly optimal dual variable (see 



Chapter 5 from Shapiro et al. (2009) for a review of the topic). In a SAA the expected value of the 
stochastic program is approximated by the sample average function over the observations {qm}m=i- 
In our case, we have that 



arg min — max{(/m — v,0} + pv. 



Equivalently, the previous problem can be stated as a linear program, and in this case one obtains 



the training-based Primal-Dual method as described in Devenur and Hayes (2009). It can be shown 
that, under some conditions, the non-parametric estimator Vm is consistent and asymptotically nor- 
mal dShapiro et al]|2009[ ). As we shall see, this estimator is not necessarily efficient. It is not hard to 



prove that the sample (1 — p)-quantile is an optimal solution to the SAA problem. Hence, from the 
asymptotic distribution of the (1 — /3)-quantile we have that 



mivM-v)^M{0,u'{9)), 



where the variance is u'(9) — -r-rzvi- 

Analysis. Both the parametric and the non-parametric estimators converge, as the number of sam- 
ples increases, to the true optimal solution. However, the non-parametric estimator is not as efficient 
as the parametric counterpart. Indeed, this is expected since the maximum likelihood estimator is 
known to be asymptotically efficient. We measure the relative efficiency as the ratio of asymptotic 
variance of the non-parametric estimator to parametric one, i.e. e{9) = 

Until now, our analysis has been in terms of the optimal dual solution. The rationale is that the 
closer the dual variable is to the true value v, the better the performance of the policy should be. Next, 
we quantify analytically how does a deviation from the optimal solution impacts the performance of 
the policy. To assess the performance of the policy we look at the fluid limit as described in Section 



EC. 4 The next proposition shows that the relative efficiency in terms of the performance is exactly 



equal to s{9). Hence, there is no loss in looking at the relative efficiency of the estimators instead. 

Proposition 8 The relative efficiency of the non-parametric estimator is 

dG . ' 



e{9)=p{l-p)m\^-^{v\9)^ >1, (EC.l) 
which is exactly equal to the relative efficiency in terms of the policies ' performance. 
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Proof of Proposition^ For (EC.l), we use that ^ip\0) = ^iv\e)/g{v\e), which follows from 



the implicit function theorem. In view of Cramer- Rao lower bound, we have that e{9) > 1. 

Next, we look at the average yield of the policy as the number of impressions grows to infinity 
when a bid price of u is employed, denoted by J{u). The limiting performance is given by 



J{u) 



pEe[Q\Q>u], iiu<v, 
Eg[Q]-{l-p)Ee[Q\Q<u], iiu>v. 



Under our assumptions, the performance function is continuous u. One would be tempted to apply 
the Delta Method to derive the asymptotic distribution of the performance. Unfortunately, J(-) is 
not differentiable at v. However, it is the case that the performance function is semi-differentiable at 
V with finite right-derivative > and left-derivative JL{v) < 0. Thus, we can apply an extension 



of the Delta Method for directionally differentiable functions proved by Shapiro (1991), and obtain 

V^{ J{vm) - Jiv)) dj(v;MiO,u'ie))^ , 
JKf) - Jiv)) dJ (^v;MiO, uie))) , 

where dJ{v;S,) is the Gateaux derivative of J at the point v along the direction ^, which is given by 
dJ{v;^) = J'^{v)^ when ^ > 0, and dJ{v;^) = JL{v)C when ^ < 0. Note that the asymptotic variance of 
the performances are u'{9) ■ K, and u{6) ■ K respectively, where the performance scale factor is given 
hy K = \{J'+{v)'^ + JL{v)^) — ^(J+(f ) — J!_(f ))^. Thus, the relative efficiency of the performance is 
identical to e{9). □ 

Examples. To fix ideas we consider two simple examples. First, suppose that Q ~ exp(0). The max- 
imum likelihood estimator is given by 9'^^'^ — (^Ji Sm=i 9™) > Fisher information is I {9) = 
9^^. The optimal dual variable isv = —9^^ In p. Hence, the relative efficiency is e{9) = (1 — p) /(pin p). 
In this case, the relative efficiency is lower bounded by £{9) > 1.544. The lower bound is tight, and 
attained at p ~ 0.2032. The relative efficiency as a function of the capacity to impression ratio is 



plotted in Figure EC.l, As shown in the figure, the relative efficiency may be arbitrarily bad as the 
capacity to impression ratio gets close to zero or one. 

For the next example we assume that qualities are normal with known variance and unknown 
mean, that is, Q ~ AA(0,fj^). The maximum likelihood estimator is the sample mean, 0™'° = 
]if X]m=M9™' Fisher Information is I {9) = . In this case the relative efficiency is given 

by £{9) = 27rp(l — p) exp ($~^(1 — p)^), with <I>^^ being the inverse of the standard normal c.d.f. Here 
the relative efficiency is lower bounded by e{9) > with the minimum attained at p = 1/2. Inter- 
estingly, the relative efficiency is invariant under monotonic transformations of any random variable. 
Hence, the previous result holds too for the log-normal distribution. 



Experiments. The previous analysis was in terms of a single advertiser; in Section 6.2 we show 



experimentally that the advantage of parametric estimation extends to multiple advertisers as well. 

Appendix EC. 2: Incorrect Assignments in the User Type Model 

In Section [5] we introduced a user-type model with good- will penalties to accommodate the fact 
that advertisers have specific targeting criteria. If the contracts are feasible, that is, there is enough 
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Figure EC.l Relative efficiency as a function of the capacity to impression ratio p for the exponential distribution, 
and the normal distribution with known variance. 



inventory to satisfy the targeting criteria; one would expect our policy to assign only impressions 
within the criteria. In this section we formalize the concept of a feasible operation, and give sufficient 
conditions under which the stochastic control policy does not assign any impressions outside of the 
targeting criteria. 

It is straightforward to state the problem of determining whether contracts can be satisfied or not, 
as a feasible flow problem on a bipartite graph. The problem can be formulated on an graph with one 
node for each user type T with a supply of tt(T), on the left side; and one node for each advertisers 
a £ Aq with a demand pa, on the right side. Then, we say that the operation is feasible if the user 
type-advertiser graph admits a feasible flow. 

The feasibility of the operation, albeit necessary, does not suffice to guarantee that no impressions 
outside the targeting criteria are assigned to the advertisers. When advertisers compete for the same 
type, and one of them obtains a potentially unbounded reward for that type; it may be optimal 
to allow the latter advertiser to cannibalize the user type, and force the others advertisers to take 
types outside of their criteria. This may occur, surprisingly, for all conceivable penalties. However, 
if qualities are bounded, and penalties are set high enough, then the optimal policy would not 
recommend the assignment of impressions outside the targeting criteria. Even in this case some 
impressions may be incorrectly assigned in the left-over regime, but the probability of this event 
decays exponentially fast. We formalize this discussion in the following proposition. 

Proposition 9 Suppose that the user type-advertiser graph admits a feasible flow, and that qualities 
and bids from AdX are bounded by miUaTa. Then, the stochastic control policy does not assign any 
impressions outside of the targeting criteria, except perhaps for the left-over regime. 

Proof Sketch of Proposition^ For simplicity we consider the case without AdX. Let i be an 
optimal solution to the DAP, and suppose that some advertisers are assigned some types outside of 
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Types Advertisers 
Figure EC. 2 Example with two user types, and two advertisers. 

their targeting criteria. We will construct another solution with greater or equal yield in which no 
incorrect assignments are made. 

An optimal solution to the DAP is a vector of functions ix^a '■ [0, 1] for a G Aq and T £T such 

that 

IEiT,a {Q)=Pa, Va G A , 

Ter 

5^ iT,a(Q)=vr(r), (a.s.) VTeT, 

We refer to each of the functions in a solution as components. 

Next, we construct a feasible solution i° to the DAP from a feasible flow of the user type- advertiser 
graph. Take the difference Ai = — i, which is a circulation in the user type-advertiser graph. The 
circulation A may have components of mixed signs. Because i° has no incorrect assignments, if 
advertiser a is assigned a type T ^ a not in her criteria, then the circulation verifies that Aix.aiQ) = 
—iT,a{Q)- Hence, the components with incorrect assignments are negative. 

Let a be an advertiser that is assigned a type T ^ a not in her criteria, that is, Efi^ ,j(Q)] > 0. We 
may find an augmenting cycle w containing the incorrect assignment, such that if we push some flow 
along this cycle, we construct another solution i + w with fewer incorrect assignments. The cycle w 
has at most ^-|- 1 positive components, and at most ^-|- 1 negative components. The cost associated 
to the negative components is at most min^/ Tq' — Tq < 0. All positive components arcs have a 
cost of at least zero, and the total cost of this cycle is positive. Thus, the new solution i + w has 
greater or equal yield. Moreover, W.[{i + w)T,a{Q)\ < IE[iT,a(Q)], and no new incorrect assignments are 
introduced. Repeating this procedure, we may construct a solution with no incorrect assignments. □ 

Note that since the left-over regime is vanishingly small in proportion to the length of the horizon 
(Cor.[T]) this implies that the number of unassigned impressions is small. Thus in practice, a publisher 
may set C'^ = Ca + e, discard any impressions assigned by the policy outside the targeting criteria, 
and ensure that contracts are filled properly. 

Next, we prove by example that the requirement that qualities are bounded is necessary for the 
previous result to hold. Consider a publisher who contracts with two advertisers, and agrees to deliver 
one half of the arriving impressions to each one of them. Additionally, there are two impression types, 
denoted by Ti and T2, each occurring 50% of the time. The first advertiser only cares about the first 
type. She obtains a reward of zero for Ti, and the advertisers pays a positive penalty r each time a T2 
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impression is assigned to her. The second advertisers admits both types, but only obtains a positive 



reward Q~exp(l) for the first type. The setup is shown in Figure EC. 2 

A feasible policy could assign all Ti impressions to the first advertiser, and T2 impressions to the 
second advertiser. However, such policy is not optimal. Notice that both advertisers compete for the 
Ti impressions, and the first advertiser could extract a potentially high quality from them. It is not 
hard to see that the optimal dual variables are Vi = — r, and V2 = 0; and the optimal objective value is 
|E[Q — t]+ = le""^. Hence, it is optimal to assign those Ti impressions with quality greater than r to 
the second advertiser. Thus, no matter the value of the penalty, a fraction e"'^ of the total impression 
assigned to the first advertiser are undesired. 

Appendix EC. 3: Computation 

In this section we describe show to compute the optimal policy for our data model. The main problem 
resides in the computation of the dual objective in Q and its gradient given a vector of dual variables. 
Objective. The first term of the objective can be written as 

ER ( max{Q, - vS] = V 7r(T)E R ( max{Q, - vS] \ T 

VT agTUTc 

VT \ aeT / 

where the first equation follows by conditioning on the type, and the second because the events are 
a partition of the sample space. Next, we show to compute the expectations lT,a{v). 

Let Mt{v) = Ta.a.yiaeAo\T{—Ta — fa} be the maximum contract adjusted quality of the advertisers 
(including the outside option) that are not in the type, and axiv) the set of advertisers that verify 
the maximum. Then, we have that 

It,o{v) = R {MT{v))¥{Qa -Va< Mt{v) Va' e T} 

= R{Mt{v))Gt{Mt{v)+Vt), 

where Gt{-) is the c.d.f. of Qt, and Vt is the vector of dual variables for the advertisers in the type. 

For a €zT, we compute the expectation by conditioning on the continuous random variable Qa- 
Further, suppose that we partition the mean vector and covariance matrix in a corresponding manner. 
That is, fJ-T = i li'-a ) ) ai^d = ^ ^1'^^ ^-a-a ) ■ instance, /i_a gives the means for the variables in 
T\{a}, and S_a,_a gives variances and covariances for the same variables. The matrix S-a,a gives 
covariances between variables in T \ {a} and set a (as does matrix S^^-q). Because the marginal 
distribution of a multivariate normal is an univariate normal, we have that Qa ~ lnA/'(/Ua, Sa,a)- We 
denote by gT,a{-) the p.d.f. of Qa- Similarly, let Q^a be the vector of qualities for advertisers in 
T \ {a}. Conditioning on Qa = Qa, the distribution of Q_q is log-normal with mean vector fi^a — 
^-aAla- fJ'a)/i'^a,a), and covariaucc matrix S-a.-a - i'^-a,a'^a-a)/i'^a,a)- Wc dcuotc its c.d.f. by 
Gr.-ai')- Putting all together, we have that 

ItA^) =E[R {Qa - Va) F{Qa' " Va' < Qa " ^^a Vo ' / ^ | Q a} \ T] 
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R{qa - Va)GT~a{qa - Va + V 

— a )QT,a 

I Va+Mrp(v) 

where v^a is the vector of dual variables for advertisers in T\ {a}. 

Gradient. The forward derivative of the dual objective can be written as 



-R\Qa-Va> max {Qa' - v'^} \ + Pa 

a£Ai,)\a 



VT 



1 - S*{Qa-Va)) l\Qa-Va> HiaX {Q a' - v[^] \ \T 



+ Pa 



E 



7r{T)PT,a{y)+Pa, 



T:aeT 



T-.a^T 
a<^arp{v) .\aj'{v)\—l 



where the contributing types for the forward derivative are those where a is in, and those where a is 
not in but verifies exclusively the maximum of the types not in [Mt{v)). If two or more advertisers 
verify the maximum Mxiv), then increasing Va does not have an impact of the type's contribution 
to the objective. When a^T, the expectation is given by 



'1 



'{Mt{v)))Gt{Mt{v) + vt) 



Similarly to the objective, when a&T we have that 

poo 

PT,a{v)= / {1- S*{qa-Va))GT,-a{qa-Va+V^a)gTAQa) dqa- 

Jva+Mj.(v) 

The backward derivative is computed in a similar fashion. The only exception is that, when a^T, 
and a verifies the maximum M7-(t;),the advertiser always contributes to the derivative regardless of 
the number of advertisers that attain the maximum. Hence, 



T:aeT 



Pa- 



T-.a^T 



Optimization. We solve the dual problem Q using a Gradient Descent Method. At each step 
the objective and its objective are computed as described previously. Notice that, when multiple 
advertisers verify a tie, the objective is not differentiable. In this case a descent direction is constructed 
using the forward and backward derivatives (if possible). 

Ties. For the following, we assume that the instance is not degenerate, that is, the variances within 
the types are positive, and no two advertisers are perfectly correlated. Then, within each type, non- 
trivial ties can only occur between the advertisers that are not in the type (we refer to the non-trivial 
ties as those in which multiple advertisers attain the same contract adjusted quality). Moreover, there 
can be at most one tie within each type, and this happens when the maximum Mt{v) is verified by 
many advertisers, that is |aT(i')| > 1. With some abuse of notation, the probability of such a tie is 
given by 7r(T)PT,aj^{v) and it should be split among the advertisers ariv). Note that the number of 
non-trivial ties is 0{T), and the tie-breaking rule can be computed efficiently by solving a feasible 
flow problem. 
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Appendix EC. 4: Fluid Limit 

Exploiting our generative model we can construct a fluid model, and obtain the limiting performance 
of an arbitrary bid-price policy as the number of impressions grows to infinity. We first describe 
the fluid equations governing the dynamics of the fluid model, then we construct a solution to such 
system, and then prove that the stochastic algorithm satisfies the fluid equation in the limit. For 
simplicity, we focus our analysis on the case with no AdX, and no ties; though, a similar analysis 
applies to the more general case. 

In the following we analyze the performance of the stochastic control policy when implementing 
some (sub-optimal) bid-prices v. Let w be a sample path, J„(a;) be the cumulative yield collected 
up to impression n, and S„^a{^) the number of impressions assigned to advertiser a up to time n. 
We extended the previous definitions for an arbitrary time, by taking their linear interpolations, 
so that they are continuous. The previous functions are random elements on C[0,oo). We shall 
construct the fluid limit by scaling capacity and time proportionally to infinity, and considering 
a continuous flow of impressions arriving during an horizon of length 1. More formally, we deflne 
Sa{t) = limi\f_^oo N~^StN,ai^), whlch can be interpreted as the fraction of impressions assigned to 
advertiser a by time t. Similarly, we deflne J{t) = limj^^oo JtN{^^) as the cumulative yield up to 
time t. We are interested in computing J(l), the total limiting yield of the algorithm under bid-prices 

V. 

When capacity is scaled, each advertiser has a capacity of pa, and the fluid model should satisfy 
the following differential equations 

J'{t)= V E[Q,l{a = argmax{g,,-?;,,}}l, (EC.2a) 

aeAit) ^ ^ 

S'^it)=F\a = aTgmax{Qa' -Va'}}, Va G A (EC. 2b) 

A{t) = {aeAo:Sait)<Pa}, (EC.2c) 



with the initial conditions 5^(0) = 0, and J(0) = 0. In (EC. 2c), A{t) is the set of advertisers that 



are yet to be fulfilled (including advertiser 0, which is fulfilled when the time comes all impressions 



should be assigned directly to the advertisers), and (EC. 2b) determines the rate at which impressions 



are assigned to each advertiser. When one advertiser is fulfilled and the fraction of impressions S'^{t) 



reaches its capacity pa, it is excluded from A{t), and its rate is driven to zero. Finally, (EC. 2a) 
determines the rate at which yield is generated. 



It is not hard to see that the solution to the fluid equations (EC. 2) is piecewise linear, and continu- 
ous. We construct a solution as follows. Let an epoch, denoted by t'^, be the time in which the contract 
of any advertiser is fulfllled (including advertiser 0). The horizon [0,1] is partitioned in consecutive 
pieces, each culminating with an epoch. The k^^ piece spans the interval [t'',t'^^^), and has a length 
of A*^ = t''^^ — . Since one advertiser is fulfilled at each epoch, there are at most A + 1 pieces. 



Let A be set of advertisers yet to be satisfied at the beginning of stage k as given by (EC. 2c), 



be the service rate for advertiser a during stage k as given by (EC. 2b), and y be the yield rate 



during stage k as given by (EC.2a). The length of a stage is determined by the advertiser that is first 
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fulfilled. Once determined, the solution is constructed recursively as follows 



^ ^-^ I Pa Sa{tk) , ^ (EC.3a) 



where a'' is the advertiser that verifies the minimum in (EC.3a), and initially t° = 0, and = Aq- 
The functions Sa and J are obtained as the linear interpolation of the values at the endpoints of 
the interval. Fortunately, the rates can be easily obtained by evaluating the dual objective and its 
gradient. Let ip''{v) be the objective in Q when restricting to the set of advertisers A''. Then, we 
have that r'' = p — VV''^(f ), and y'' = ijj^{v) — v • ViIj''{v). 

We conclude by showing that functions obtained are actually the fluid limit of the stochastic 
process induced by the algorithm. 



Proposition 10 The fluid limits Sa{t), and J{t) are a solution to the fluid equations (EC.2). 



Proof. Consider the sequence of random elements S^^aitjUj) = N ^StN,a{i^), and J}^{t,uj) 



N ^JtN{^)- We would like to show that the previous sequences are tight. By Theorem 8.3 in Billings- 



ley (1968), a sequence of random elements {X^} in C[0,oo) is tight iff (i) {Xn{0)} is tight, and (ii) for 
all e > and ry > 0, there is a (5 > and an integer Nq such that ¥{sup^^^,^^^^ {X^lt') —XN{t) \ > e} < r/ 
for N>Na. 

The first condition is trivially satisfied for both sequences. Disregarding integrality issues, which 
are not important in the analysis, and using the fact that the processes are non-decreasing, we have 
that 

sup \SN.a{t') - SN.a{t)\ < {S(t+S)N.a " StN,a) < S. 
t<t'<t+S 

Thus, by picking 5 <e the second condition is satisfied for the number of impressions assigned. For 
the yield processes, employing Markov's inequahty, and the bound Ei?^ < ^^maXa{EQ^} for the 
yield in any single period, we obtain 



6 



n=tN 



^^ A^max,{EQg} 



which can be bounded from above by rj by picking a small enough 5, and N > 1/S. 

Next, we show that the fluid limit converges to the solution of the equation (EC.2b). Before 
proceeding we state some definitions. Let the stopping time n% = inf{n : Sn,a > C'a for some a G A'^^} 
be the time in which the contract of the k*"^ advertisers is fulfilled. In our previous terminology, 
is the fc*^ epoch and the beginning of the A;*'' piece. Similarly, we let A% = {a£ Ao : S^i^ ^ < Ca] to 
be the set of advertisers that are active during the k^^ piece. The initial values are given by n% = 1 
and A'It = Aq. 
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We intend to show that N~^n% ^ A%^A'', and N'^S„k ^ Sait'') as N — 7- oo in an almost 
sure sense. We proceed by induction in k. The base case follows trivially. Suppose that our claim 
holds for k, we intend to show that it holds for k + 1. By the definition of the {k + iy^ epoch, it should 
be the case that 5^fc+i_^ ^ < Ca for all a G A% and -S^fc+i ^> Ca for exactly one advertiser a G A%. 



'AT ^ "JV 

Notice that the number of impressions assigned to ad a up to the stopping time can be written as 



where the summands on the right-hand size are i.i.d. bernoulli random variables with success prob- 
ability p|a = argmax^/g_4fc^{(5Q/ — Va'}!- From the induction hypothesis, together with the Trian- 
gular Strong Law of Large Numbers, we get that (EC. 4) goes to Sa{t'') + (lim A^^^n^"^^ — t'')r^ 
since the success probability converges to r^. It is not hard to see that the same limit holds for 
N~^S k+i , . Thus, we have that for all advertisers yet to be satisfied it should be the case that 
Saif") + (lim7V-in^+^ -t^^Or^' < pa, and for at least one advertiser 5'„(t'=) + (limiV-in^+^ - t'=)r^ > pa- 
Combining this expressions we get that in the limit the stopping time should satisfy 

hm ^=t'+min( B^^^MA ] = ^^+1 . 
N aeA^X / 

In turn this implies that N~^S^k+i ^ — )• Sa{t''^^) and — ^ A'''^^, thus concluding the inductive 
step. 



